Daniel R. Andrews 


Shooting the Moon 

This is an unusual book about an unusual mission to find water on the moon. 

The LCROSS mission had a noble purpose, but with little time and money, we 
had to do things differently. 

My goal with this book is to let you inside the project to see the world through the 
eyes of the project manager. How we did what we did. Why we did what we did. I 
suspect you’ll find lessons that apply to project work anywhere. 

The stories within come from the many notes I took over the course of the 
mission. They include my personal reflections, observations, assessments, 
learning, techniques, and recommendations. The chapters are sometimes 
analytical and procedural, and at other times observational, reflective, amusing, 
and absurd. 

It was a great honor to lead the LCROSS team - we all learned and played harder 
than we probably thought we could. We became a force and a pathfinder, all 
while forging friendships that will last a lifetime. On behalf of the LCROSS 
team, I hope the stories and learning captured within this book are valuable and 
amusing to others. 
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Take calculated risks. 


Be willing to change course. 


Keep moving. 
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LCROSS NASA-ARC first team picture, 2006. Last row standing (L to R): Rich Aros, Tony 
Ricco, John McCormick, Dana Lynch, Leonid Osentinsky, Ken Galai, Rusty Hunt, Jim 
Hanratty, Brian Day, Silvano Colombano, Tom Luzod, Darin Foreman; First row standing (L to 
R): John Bresina, Paul Tompkins, Kim Ennico-Smith, Arlene Schwartz, Rosatina Chan, Bob 
Barber, Dave Squires, Steve Ord; First row kneeling (L to R): Leonard Hee, Tony Colaprete, Gil 
Kojima, Jeff Brown, Mike Schobey, Daniel Andrews, John Marmie, Roger Arno. 


Someone told me early on, “You ought to take a team picture at the beginning of this project and then 
take another one at the end, because you ’re going to see a lot of changes between now and then. ” This 
person was referring to the fact that we were about to embark on a very challenging adventure, one that 
would consume some of us (who wouldn’t be able to see it to the end) and age the rest of us. So here is 
the first NASA-Ames Research Center (NASA-ARC) team picture, taken near the beginning of the 
project. At the end of this book is the final team picture. 
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Chapter 1 — Introduction 


This story is about an unlikely NASA mission to the 
Moon. It was unlikely because it was started with far too- 
little time and too-little money to complete. It was 
unlikely because it was able to take chances - to accept 
risk of failure. It was unlikely because it was searching 
for the unthinkable: water-ice on the moon... 

The mission of the Lunar CRater Observation and 
Sensing Satellite (LCROSS) was to investigate the 
possibility of water ice in craters on the Moon’s poles. 

This is certainly an interesting scientific topic in itself, 
but I intend to focus on the compelling experience of 
managing the LCROSS Project in the context of this storied Agency. Perhaps most interesting are the 
implications this story has for managing any development effort, lunar or not, and working a balance to 
achieve success. 

NASA is by design a risk-taking agency within the US Government. It could be argued that NASA’s 
purpose in the aerospace community is to take on the really big challenges that either the corporate world 
can’t afford, are not yet profitable endeavors, or are just too risky for private corporations to entertain. 
However, expectations of the Agency have evolved. A combination of grim human tragedies and some 
very public cost and schedule overruns have challenged the public’s and Congress’s tolerance for risk- 
taking within the Agency. NASA, which is supposed to be in the business of taking risks to do bold, 
difficult things, has become less and less able to do so within its cost framework. Yet effectively 
replacing prudent risk management with attempts to “risk-eliminate” is completely unaffordable. 

So where does risk-taking fit within the Agency, or within private/corporate organizations for that 
matter? Where astronauts play there is clearly concern about risk. When an organization puts humans in 
harm’s way, it is understandably going to take extra effort to assure nobody gets hurt. Doing so, of 
course, costs money - a lot of money to pay for labor and hardware which is attempting to assure nothing 
will go wrong. Sophisticated designs, with doubly- or triply-redundant systems, extensive testing to 
verify those systems, and numerous engineering test units built to learn and evolve a hardware design, all 
drive the cost and time required to implement. Human spaceflight is an expensive business because of 
the exceptional system complexity and levels of assurance required for human space travel. 

What about missions that do not involve human spaceflight? What about missions whose potential 
failure will not take a human life, whose costs are small and whose urgency and importance are limited 
by design? A portfolio consisting of this type of mission can be designed to be risk tolerant, not requiring 
large expenditures to guarantee against failure. With the money saved, the number of missions that can 
be executed within the portfolio grows, or the total cost of the portfolio can be reduced. The NASA 
LCROSS mission is a pathfinder example of a low-cost, quick tum-around mission which struck a 
balance on mission risk, while accomplishing big objectives, like defining how we understand the Moon. 



Figure 1-1: LCROSS Mission. 
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Chapter 1 — Introduction 



Figure 1-2: Image from the classic 1902 film, La 
Voyage dans la Lune ("A Trip to the Moon"), by French 
magician/filmmaker Georges Melies, representing one 
of the world's first major science-fiction films. 


Ultimately, LCROSS is a story of a 
collection of groups brought together in a 
project context to walk a shared road on a 
mission to the Moon. Each party had reasons 
to play. The Agency wanted to demonstrate 
an effective way to make use of excess 
launch capability and to work cheaply; 
NASA-Ames Research Center (ARC) 
wanted to prove it was able to run small, 
fast-paced, lightweight missions; Northrop 
Grumman Corporation (NG) wanted to show 
that it could execute agile, quick turn-around 
spacecraft builds on budget and schedule; 
the commercial instrument sector found an 
on-ramp to space and lunar applications, 
which could propel their businesses into a 
new market. One of the great successes of 
LCROSS was aligning each team member’s 
needs into a common purpose which upon 
success benefited everyone involved. 
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Chapter 2 — In the Beginning... 


Early in 2006, NASA’s Exploration Systems Mission Directorate (ESMD) held a competition 1 for 
NASA Centers to come up with innovative lunar mission ideas to launch with the Lunar Reconnaissance 
Orbiter (LRO) to the Moon: 

“ESMD is seeking mission candidates with total costs in the $50M range. Strong consideration 
will be given to lower cost mission proposals. Missions over $50M will be considered, however, 
in no case will NASA consider missions with a total cost exceeding $80M. ” 

In addition to the financial constraint, the selected mission would have to be ready to make the LRO 
launch date in 31 months, could weigh no more than 1000 kg, and (here’s the interesting part) would be 
designated a “Class D” mission. We’ll discuss the “Class D” part later, but the Agency was in effect 
issuing a fixed-price mission on the winning team. This provides a sense of where the priorities were to 
lie for LCROSS: cost and schedule were capped, so those constraints could not be violated. The “Class 
D” criterion was, in short, a statement of relative risk tolerance for the mission - a pretty high risk 
tolerance as it turned out. In fact, the requirements later developed for the LCROSS project included 
“Minimum Success Criteria” and “Lull Success Criteria,” which helped to quantify mission priorities. 



The opportunity to place a secondary payload on LRO led 
NASA-Ames Research Center (ARC) at Moffett Lield, CA, to 
embark on a brainstorming effort termed “Blue Ice,” wherein a 
small team was chartered with originating ideas to put forward in 
the competition. Blue Ice was a rich exploration of ideas, 
missions and means that would work within the stated 
constraints. LCROSS was one of six proposals submitted by 
NASA-ARC, and one of nineteen proposals ESMD received 
from around the Agency. Lor the LCROSS proposal, NASA- 
ARC partnered with Northrop Grumman (NG) in Redondo 
Beach, CA, who wanted to demonstrate they could build an 
Figure 2-1: LCROSS official logo. inexpensive spacecraft in a short amount of time. NG would 

design, build and test the spacecraft, while NASA-ARC would 
manage and design the mission, design and develop the payload instrument suite, perfonn the project 
systems engineering, and lead mission and science operations. 


After a two-month ESMD evaluation by the Robotic Lunar Exploration Program (RLEP), I received a 
call from the RLEP deputy program manager, Butler Hine, asking me some curious questions: 


♦ Butler: “Dan, can you clarify some things on your resume? ” [Now this was strange. The program 
office calling the PM of one of the proposals and asking about resume details?. . .] 

♦ Dan: “Ummm, sure; what’s this about?” 


♦ Butler: “Well... we just want to make sure we understand everything correctly.... [I answered his 
clarifying questions and then he asked], “Can you be present in Washington DC next week? ” 

♦ Dan: “Butler... am I to interpret something from this line of questioning? ” 
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♦ Butler: “ Nope ... we ’re asking all the down-selects if they can be there... I’ve been told I can ’t tell 
you more. ” 

♦ Dan: “OK... yes, I can be wherever you need me to be.” 

♦ Butler: “ One other thing, Dan: can you wear a NASA polo shirt when you arrive? ” 

♦ Dan: “What?” 

♦ Butler: “Humor me - 1 have to ask...” 

♦ Dan: “Very good. I will be in DC on Tuesday morning wearing a NASA polo shirt.... ” 

The deliberateness with which these 
questions were asked was awkwardly 
funny on both ends of the phone call. I 
later found out the ESMD Associate 
Administrator (AA) was planning on a 
competition-style “reveal” of the winner in 
front of the press and on NASA-TV, and 
wanted team leads present for this exciting 
announcement! On the trip to DC, I 
walked into the NASA Headquarters (HQ) 
press/briefing room and was greeted by 
Butler. With a relieved look on his face, he 
promptly asked me, “So you know why 
you’re here, right?” I replied, “I think so, 
but would like to hear it from you,” at 
which point he told me LCROSS had been 
selected... a moment I won’t soon forget. 

I’ll skip through all the nerves, excitement and terror of learning of the LCROSS selection just a couple 
hours before we’d go live with TV and the press. That press event went just fine. Of more importance was 
my discussion with the AA, Scott “Doc” Horowitz, while we were waiting to go live, poised in our NASA 
chairs, surrounded by lunar globes and other Exploration trinkets to complete the set. I found him to be 
very welcoming, but also purposeful. He was especially excited about the LCROSS mission because it “is 
not your father’s NASA,” and added that there is a place for the weight and conservatism of traditional 
NASA missions, namely manned spaceflight. But he needed a way to accomplish tactical missions 
inexpensively, given the existing financial obligations the Agency had with everything from the then-new 
Constellation program to the infrastructural demands of the Shuttle Program and the International Space 
Station (ISS). He found LCROSS to be an exhilarating mission, one that would inspire the public as well 
as ascertain whether there is water ice on the Moon. Most important to him, however, was the opportunity 
to establish a cost-effective way to execute meaningful missions on a budget. 



Figure 2-2: Scott "Doc" Horowitz, Daniel 
Andrews, and Butler Hine at LCROSS press 
conference, in Washington DC (from left to right). 
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nsing 


1 could triple the cost to try to 
guarantee no failure or I could 
do three projects and even if 
one fails, I still get more done. 


Doc would later indicate LCROSS was a pathfinder project to see if it is possible to make practical use of 
excess launch capacity, while staying within tough cost and schedule constraints. The opportunity of excess 
launch capacity occurs when a given mission launch isn’t 
making use of all the mass that could be delivered to orbit 
on a given launch vehicle - in effect, another mission could 
join the ride without requiring additional launch vehicle 
capabilities. He specifically noted that the Agency will 
increasingly rely on smaller, high-leverage, cost-capped 
missions. He wanted me to track all that we learned through 

the process, from working with NASA’s Procedural Requirements (NPRs) to means of acquisition, and to 
detennine how and if the program office and HQ get in the way of success. He had a clear understanding of 
how and where this type of mission fits within the NASA portfolio. As he stated in a later interview, “I could 

triple the cost to try to guarantee no 
failure or I could do three projects 
and even if ones fails, I still get 
more done. 2 ” So between the 


original call for the proposals and 
this dialogue with the ultimate 
stakeholder for the mission, the 
LCROSS context was clear: cost 
and schedule are key drivers. . .and 
the Associate Administrator is 
willing to take risks. 


lations 


ater Observation & 
tellite (LCROSS) Team 


?Y 


ON DA 


PA 


As I look back on this project and 
all the different members of the 
team, I see that LCROSS evolved 
to be much more than just an 
experiment for the AA. This was 
Figure 2-3: LCROSS congratulatory banner at NASA-ARC. about creating a new way to 

execute missions and to fill the 
tool bag of approaches for the 
Agency. In doing so, it influenced 
protocols and approaches both 
within and outside of NASA. 


End Notes 

1. 2006-01-26: Initial NASA-ESMD Center Call, soliciting proposals. 

2. “Inside NASA’s Plan to Bomb the Moon and Find Water”, Popular Mechanics, Michael Milstein, September 2008. 
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Chapter 3 — LCROSS Mission Overview 


The LCROSS mission was to be the first in-situ study of the pristine, permanently-shadowed, polar 
lunar craters with the following exploration objectives: 

1) Confirm the presence or absence of water ice in a permanently shadowed region 

2) Identify the nature of the hydrogen signatures previously detected at the lunar poles 

3) Determine the amount of water, if present, in the lunar regolith (soil) 

4) Determine the composition of the lunar regolith 

The scientific basis for the LCROSS mission has roots in the Clementine (1994) and Lunar Prospector 
(1998) missions, which did complementary forms of resource mapping. This mapping led the lunar science 
community to conclude there might be water ice trapped in permanently-shadowed craters on the Moon. 

The Clementine Mission 1 was launched on Jan. 25, 1994 from Vandenberg Air Force Base in California 
aboard a Titan IIG rocket. It was a joint project between the Strategic Defense Initiative Organization in 
Washington and NASA, and the objectives included making scientific observations of the moon. 
Clementine's instruments were designed to assess the surface mineralogy of the moon, obtaining lunar 
altimetry and imagery from a fixed altitude. 

The Lunar Prospector Mission 1 was launched on Jan 7, 1998, aboard a Lockheed Martin solid- fuel, three- 
stage Athena II rocket, as part of the NASA Discovery Program. It was managed out of NASA’s Ames 
Research Center, with Lockheed Martin as the prime contractor. The 19-month mission was designed for 
a low polar orbit investigation of the moon, including mapping of surface composition and possible polar 
ice deposits; it completely covered the lunar surface twice a month. Originally, the mission was to have 
ended with the spacecraft crashing into the moon when its fuel ran out; however, as the mission neared its 
end the suggestion was made to use the crash as part of an experiment to investigate the possibility of 
water on the Moon. The spacecraft was successfully directed into a crater near the lunar South Pole, but 
the impact plume was not significant, probably because of a poor impact angle and low spacecraft mass. 

These two missions were the basis for the question of the presence of lunar water ice. In particular, the 
Lunar Prospector mission measured elevated hydrogen signatures in permanently-shadowed craters on 
both the North and South poles of the Moon. In light of these data, the science community pondered the 
possibility that this elevated hydrogen signature could indicate the presence of water ice, trapped just 
beneath the regolith surface of the crater floors. If water exists on the moon, it likely arrived the same 
way it did on Earth - though billions of years of bombardment by meteors and comets. Since the moon’s 
gravity is less than one fifth of Earth’s gravity, the moon has practically no atmosphere and any 
depositions on its surface are subject to direct exposure to the vacuum of space. Furthermore, daylight 
temperatures reach 250° F (121° C), and any exposed water ice will turn into water vapor and be lost to 
space. Water ice can only exist protected from these environments, and the lunar poles may provide 
exactly that. The sun never rises above certain crater rims and therefore the crater floors may not have 
seen sunlight for billions of years. With temperature estimated to be near -328° F (-200° C), these craters 
can be a “cold trap,” capturing most volatiles. 
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If the Moon has water ice in sufficient quantities, it would represent a very 
compelling rationale for future exploration and lunar outposts could be located 
in the vicinity of this invaluable resource. Water ice, after all, could be 
converted to consumable water, breathable oxygen, and rocket fuel, and 
potentially even serve as a means for construction when combined with regolith 
or as shielding from solar radiation. Delivering mass to the moon is incredibly 
expensive and water is very heavy for a small volume. Delivering a Vi liter 
bottle of water to the Moon is projected to cost a minimum of $15,000 by 
weight. Therefore, having this resource available in situ to future explorers and 
inhabitants of the Moon is clearly worth investigating. 



LCROSS planned a clever way to conduct its experiment of confirming if there 
is water ice present on the moon. In short, we cheated - well sort of. This NASA 
LRO-comanifesting competition levied a mass constraint of 1000 kg on all 
competitors, as this was all the mass available after LRO’s needs were met. The 
cheat that LCROSS employed was to get a total mission mass three times that of 
the 1000 kg limit, by making use of the upper stage of the Atlas- V rocket (the 
“Centaur”), normally space junk after delivering a payload. The Centaur 

brought 2300 kg of 


Figure 3-1: 1/2 liter 

bottle of drinking water 
would cost >$15,000 to 
bring to the moon. 


mass to the LCROSS 
mission “for free” in 
addition to the 1000 kg 
allotted to the secondary payload. Since the 
Centaur mass is accounted for in the launch 
vehicle budget, not the LCROSS mass budget, we 
really did live within our 1000 kg allotment. 


Figure 3-2: Artist concept of LCROSS separating 
from Atlas (Image Approved for Public Release; 
Susan Cohen, Blake Bullock, NGST 1-29- 08; 
Image Source: Northrop Grumman) 


So what did we do with the Centaur? The short 
story is we used it as a lunar kinetic impactor. 
LCROSS “dropped” that Centaur (which weighed 
about the mass of a large sports utility vehicle) 
into one of those permanently-shadowed craters 
at an estimated 1.5 miles per second (2.5 km/s) - 
three times the speed of a bullet, in order to kick 
up the material from the crater floor. The 1000 kg 
LCROSS “shepherding spacecraft” collected data 
about the debris plume using the nine on-board 
science instruments and transmitted the data back 
to mission control before impacting the surface 
itself, just four minutes after the Centaur. 
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The longer story is that the Atlas launch vehicle is made of a booster stage and the Centaur upper stage. 
The LCROSS spacecraft was mounted atop the Centaur with the LRO spacecraft mounted atop LCROSS. 
On June 18, 2009, both were launched aboard an Atlas-V rocket from Cape Canaveral, FL. Once the 
Atlas-V achieved the LRO lunar insertion requirement, LRO was separated, enabling it to move forward 
on its mission, leaving LCROSS and the still-attached Centaur behind. The Centaur then performed a 
series of venting maneuvers to eliminate gasses which could contaminate the impact measurement. The 
Centaur, now inert and non-functional, was ready to be the LCROSS mission lunar impactor. 
Approximately five days after launch, LCROSS 
performed a swing-by of the moon, entering into an 
Earth polar orbit. This Earth-orbiting cruise phase of 
the mission took a little more than three months, 
before beginning the terminal phase of the mission: a 
collision course with the moon. Meanwhile, this 
long, looping LCROSS orbit around the Earth 
allowed the Lunar Reconnaissance Orbiter, now in 
orbit around the moon, to commission its 
instruments and to collect data about the South Pole 
craters; this data helped the LCROSS science team 
refine its impact target crater selection. During that 
cruise phase the Ops team executed several 
trajectory correction maneuvers (TCMs) to provide 
for the final lunar approach required for the impact. 

The last TCM positioned the Centaur upper stage for 
ballistic lunar impact. The Centaur and the 
shepherding then separated, about nine hours before 
impact, followed by the shepherding spacecraft 
performing a braking maneuver by firing four 
thrusters for a few minutes. The purpose for this 
braking was to enable the released Centaur to get to the Moon first, and to give the shepherding spacecraft 
time to observe the ejecta rising from the site of impact. We estimated that the Centaur’s impact would 
excavate 250 metric tons of regolith, leaving an impact crater approximately 92 feet (28 m) in diameter by 
16 feet (5 m) deep. Most of the excavated material or ejecta would be thrown upward at a velocity of more 
than 820 feet per second (250 m/sec), some of it sent several miles upward, but could vary depending on 
the makeup of the floor of the crater; remember the nature of these crater floors in unknown. As the 
shepherding spacecraft continued its delayed descent, the cameras and sensors in the instrument suite 
measured the constituents of the ejecta plume, observing and measuring all the way down to its own 
inevitable impact on the Moon four minutes later. The sensors of other lunar-orbiting and Earth ground- 
based telescopes, such as LRO itself, Hubble Space Telescope, and Earth Great Observatories in Hawaii, 
Chile, etc., also monitored the impact, with varying degrees of success. 



Figure 3-3: Artist concept of Atlas upper 
stage lunar impact with LCROSS 
shepherding spacecraft looking on. 
(Image Approved for Public Release; 
Susan Cohen, Blake Bullock, NGST 1-29- 
08; Image Source: Northrop Grumman) 


12 



Chapter 3 — LCROSS Mission Overview 


The LCROSS science payload consisted of nine instruments: one 
visible, two near-infrared and two mid-infrared cameras, one visible 
and two near-infrared spectrometers, and a photometer. A data 
handling unit collected the information from each instrument for 
transmission back to LCROSS Mission Control at NASA-ARC. 
These instruments were envisioned back in the LCROSS proposal 
as low-cost, rugged, commercially available components... for an 
Earth environment. However, to be sure the instruments survived 
both the launch environment and our time in space, the LCROSS 
payload team put them through rigorous testing that simulated 
launch and the conditions of space. In some cases that testing 
revealed weaknesses, and the team worked with the manufacturers 
to strengthen their designs for the LCROSS application. 

The LCROSS instrument payload was designed to provide 
mission scientists with multiple complementary views of the 
debris plume created by the Centaur impact. As the debris plume 
rose above the target crater’s rim, it was exposed to sunlight, and 
water ice, hydrocarbons and organics vaporized, breaking down 
into their basic components. 



Atlas V 401 


Figure 3-4: LRO/LCROSS 

fairing configuration. 


LRO/LCROSS Launch Configuration 
Atlas V 401 


Fairing 
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Figure 3-5: Exploded view of the LRO/LCROSS 

spacecraft launch configuration. 


These molecular components 
were monitored primarily by 
the visible and infrared 
spectrometers. The near- 
infrared and mid-infrared 
cameras determined the total 
amount and distribution of 
water in the debris plume. The 
spacecraft’s visible camera 
tracked the impact location 
and the behavior of the debris 
plume, while the visible 
photometer monitored the 
intensity of any flash created 
by the kinetic energy released 
during the Centaur impact. 
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The LCROSS mission was a clever solution to the LRO secondary payload opportunity. It maximized the 
trade space of the given limitation of mass, but as will be discussed later, also made astute use of the 
schedule constraint and most importantly, the project’s assigned risk position. 


End Notes 

1 . NASA LCROSS Press Kit, 2009. 
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Chapter 4 — Faster, Better, Cheaper? 


In the Tony Spear-led NASA FBC [Faster, Better, Cheaper] Task Final Report , the origins of FBC are 
addressed 1 . FBC was originally touted by Dan Goldin, NASA Administrator in 1992. Cost pressures and 
recent mission overruns were seeding the desire to change how projects were executed within the 
Agency. Change was brewing, countering the weight and heft of typical NASA missions. There was a 
desire to shorten development times, reduce cost, and increase the scientific return by flying more 
missions in less time. In 1992, NASA created the Discovery Program, with a focus on creating 
competing project teams with specific focus on some facet of our solar system, at a $150M cost cap per 
mission. One of the tenets of this program was to, “...achieve outstanding results by launching many 
smaller missions using fewer resources and shorter 
development times f 2 and be ready to fly within a few years 
of selection. The first three Discovery Program missions, 
selected over several years, were NEAR (Near Earth 
Asteroid Rendezvous), Mars Pathfinder, and Lunar 
Prospector (LP) 3 , with LP being one of the inspirations for the LCROSS mission. Tony Spear was the 
project manager for the second FBC mission, Mars Pathfinder, a $270 M mission, which was highly 
successful. In July 1999, Goldin asked Tony to “undertake a personal study of the Agency’s 
implementation of FBC,” 4 on the heels of some successes and significant failures in its application. Tony 
was to assess FBC best practices through a series of interviews and workshops throughout the Agency, 
including NASA Headquarters, ten NASA Centers, industry, and academia. NASA Administrator Dan 
Goldin said of FBC missions, “It’s OK to fail!” 5 ... /Source/ familiar to the LCROSS context? This matches 
the risk-tolerant position assigned to the LCROSS Project. Tony Spears clarifies that this statement was 
made “to encourage project managers to step up bravely to difficult, risky, but potentially highly 
rewarding missions. Failures here can be honorable, even if still traumatic to the project team.” 6 He later 
suggested that a success rate of 8 out of 10 should be a goal. Given the apparent similarity between the 
NASA FBC approach of the past and the context in which LCROSS was to operate, it’s worth digging in 
a little further on this topic. 

Not unlike LCROSS, these missions had to be quick turn-around and live within much smaller budgets 
than typical NASA missions, so the stakeholders were willing to look to new approaches to project 
execution. The performance of these three early missions provided credibility to the FBC approach, as 
NEAR, Mars Pathfinder, and LP met or beat expectations of project officials and the Discovery Program. 
This success in turn led to expanding FBC throughout the agency on much bigger projects and 
endeavors, including the Space Shuttle and the International Space Station, as congressional budget 
pressures on the Agency continued to grow. The era of billion-dollar flagship missions was being 
challenged, further leading the Agency to explore less expensive projects in other places. Of course, to 
get those smaller missions to do meaningful science at a sporty pace, NASA would need to embrace risk 
tolerance, as opposed to risk elimination. When some later notable mission failures caused 
embarrassment to the Agency, however, the FBC philosophy came into question. Teams working FBC 
projects started voicing concerns about the pace and scope of these increasingly challenging missions, 
feeling they were simultaneously pushing on cost and schedule reduction, while doing more (“ better ”). 
Can risk be managed to an acceptable level under FBC? 


NASA Administrator Dan 
Goldin said of FBC missions, 
"It’s OK to fail!” 
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During the time of the LCROSS mission, the Agency appeared to embody two cultures: the big, risk- 
averse missions, and the smaller risk-tolerant ones, with the Agency’s protocols and procedures clearly 
aimed at avoiding risk of mission failure. The type of risk we are talking about is important to understand 
because it helps providing ranking of relative risk concerns. The big risk that is generally alluded to in all 
these discussions is Mission Risk - the loss of mission. This generally equates to technical risk, as 
opposed to programmatic risks. This is important to differentiate because risk of cost overrun or schedule 
overrun has not yet reached the same level of political harm in the Agency as the politics of mission 
failure... although there are signs the politics of cost and schedule overruns is growing in importance. 
This risk aversion context made LCROSS team’s job more difficult, since it had a special set of 
constraints wherein cost and schedule were hard constraints under the threat of project termination 
should they be exceeded. 

So was LCROSS just another Faster, Better, 

Cheaper mission? To answer this, we need to 
explore the LCROSS approach more closely. On 
any project activity, there are three “buckets” from 
which a project manager must work: the cost 
bucket, the schedule bucket, and the requirements 
bucket (see Figure 4-1). Think of these buckets as 
containing the levels of project reserve or margin 
you have for each of these commodities. The cost Fi 9 ure 4 ' 1 : The Three Management “Buckets.” 

bucket holds your cost reserves; the schedule bucket holds your schedule slack; the requirements bucket 
holds a combination of technical perfonnance margins and the requirements you could trade if needed. 
These commodities are the trade space from which you manage the realities of executing a project. 
Understanding the nuance of the relative fill-levels of the buckets, and that they are not independent of each 
other, is an important project manager skill, as the fill-levels relate directly to the project risk position. 

It should go without saying that full buckets are better than empty buckets. Full buckets mean plenty of 
reserves and therefore many different means for addressing risk. An empty bucket means you’ve got 
nothing left to give for that commodity. For example, an empty schedule bucket means you have no 
schedule slack left. If a critical path item slips your schedule when you have an empty schedule bucket, 
you will not make your schedule requirement. 

There is, of course, no such thing as a no-risk endeavor. Even if you believe you’ve licked all known 
risks, there are still unknown risks lurking. So in managing a project, how daring should you be? How do 
you manage your relative risk? If your requirements are not very complex or have “been done before,” 
you may allow the bucket levels to get pretty low, effectively acknowledging that you have a low-risk 
task and can tolerate running with low margins in order to save money. On the other hand, if you’re doing 
something that has never been done before, working in uncharted waters, you’d better have some buckets 
fairly full. 
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Dissecting the FBC paradigm is fairly straight-forward. The “Faster” part of the mantra means your project 
meets or beats the required schedule; the “Better” part means you meet or beat the requirements or 
expectations you signed up to; the “Cheaper” part means you pull off the first two within your allotted cost. 
How does this relate to the buckets? It means you can’t let the schedule bucket run dry or the project will 
not be “Faster.” You can’t let the requirements bucket run dry because then the project will not be “Better.” 
You cannot let the cost bucket run dry because then the 
project will not be “Cheaper.” So if you’re managing a FBC 
project, you have to keep pretty healthy levels in all of those 
buckets, because you are pushing the limits in each of those 
commodities. In short, Faster, Better, Cheaper projects are 
not risk tolerant, since they are simultaneously pushing the 
limits on cost, schedule, and perfonnance. 


Faster, Better, Cheaper 
projects are nqt risk tQkraffit, 
since they are simultaneously 
pushing the limits on cost, 
schedule, and performance. 


The Spear report noted that in the wake of some of the notable failures of the FBC era, there was healthy 
debate on how FBC should really work. “Should the ‘Better’ go before ‘Faster’ and ‘Cheaper?’” Or the 
perennial classic, “Pick two, but you can’t have all three.” In the end, the report concluded that, “...it is 
not true that only two of three in FBC can be obtained,” specifically citing NASA-ARC led Lunar 
Prospector as a great example. As the report states, “It was certainly ‘Faster’ and ‘Cheaper’, and its 
‘Better’ was the ingenious simplicity of its spacecraft system design to make such important 
measurements.”^ h, there you have it. So this definition of Faster, Better, Cheaper, was actually intended 
to mean “Faster, Simpler . Cheaper.” Or put another way, the trade is whether the system will be a 
technical marvel, or simply good enough to do the job . 7 The report further clarifies that defining the 
appropriate scope for a FBC mission is critical to improving its risk position. That qualitative definition 
of what makes a “better” mission is very important. 


The early FBC projects had mission scopes that fit fairly well within the cost caps, good examples being 
Clementine, NEAR, Mars Pathfinder, Mars Global Surveyor, Lunar Prospector and Stardust. However, 
as the FBC philosophy started to spread within the Agency, subsequent missions became too aggressive, 
raising the bar too high, thereby incurring a fair amount of risk. The buckets didn’t have enough reserves 
to cover systems that were not “simple.” Those later missions did not have a good match between the 
effort required to be successful and the time and cost required to be successful. These were ambitious 
missions with decreasing development times that couldn’t allow for the necessary levels of testing. The 
results were oversights that could have been mitigated if the schedule, cost and performance 
requirements had been better aligned. This is where the application of FBC broke. The goals of these 
second generation missions were impressive in their sophistication, but this sophistication pulled them 
away from the idea of “simpler.” Complex designs not only have less heritage by definition, but also 
require more time to test to assure requirements satisfaction. These teams of hard-working engineers, 
technicians, scientists and project managers set off on reasonably risky ventures from the start, with little 
left in the buckets to fully test functionality in an environment where in reality, it was not “OK to fail.” 
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In retrospect, things played out fairly predictably. The context the Agency gave these Faster, Better, 
Cheaper missions was inherently at odds with the project constraints. The project teams were self- 
conflicted, with little room for contingencies, likely putting in a lot of “free” work to keep schedule and 
cost buckets in check, unwittingly growing their technical risk along the way. 

LCROSS was not a Faster, Better, Cheaper mission; 

it was a Faster, Good-enough. Cheaper mission. L( ROSS W&S not 21 dSlll. Lit til. 

LCROSS was “Faster” because of its ambitious Cheaper mission; it was a Faster, 

schedule. One could also argue that it was Good-enOUgh, Cheaper mission. 
“Cheaper” because of the cost cap and frugal 

funding profde; however, it was not a “Better” mission. As upcoming chapters will show, LCROSS 
always strived to be a high-heritage, low-complexity, and just good-enough mission. The spacecraft was 
simple by design. The technical design was intentionally a low-risk approach. Early in the LCROSS 
project execution someone said, “LCROSS is not about maximum performance, but cost containment.'’’’ 

The first step when fonnulating a project is to make sure the “bucket context” is right for the mission 
context. Are your baseline bucket levels appropriate for your stakeholder/customer risk position? Are 
stakeholders expecting a risk-free development, but not providing enough programmatic margin to address 
the issues that will certainly surface? Is the project’s risk position appropriate to its bucket levels? The real 
lesson here is to make sure the mission you are being asked to execute is in good alignment with the 

stakeholders’ expectations. A simple, cheap mission 
IX ROSS was not about maximum can tolerate more risk. A complicated, expensive 
performance, but cost containment, mission likely cannot. What is expected of you? 

LCROSS attempted to assure the right balance 
across the project buckets. Managing the cost, schedule, and requirements buckets, including deciding 
when to pour a little out of one bucket to preserve another bucket’s level, is the challenge. An example 
would be when you expend some of your mass margin to stiffen a structural member in order to preserve 
an instrument’s pointing accuracy. You need to assure “bucket parity” across the three buckets, 
commensurate with your mission context, which doesn’t necessarily mean that all buckets need to be at 
the same fill-level. 

At project selection, LCROSS was carrying 15% project cost reserves. By NASA standards, this was half 
of what is traditionally held at project start-up. LCROSS was also cost-capped, so if we ran through that 
15% reserve, we were subject to project termination. The mission carried schedule slack to the LRO 
launch date at ATP, but it was not generous for a mission that had to make launch or suffer the fate of 
being flown “dead.” As it turns out, LCROSS actually began growing its schedule reserves after ATP, 
effectively raising the fill-level of the schedule bucket; this increasing reserve will be discussed more in 
the section on managing the schedule. LCROSS ’s requirements were conceptually defined in the original 
proposal and effectively approved when the project was given the green light. Our Class D designation 
and “minimum success criteria” enabled us to trade requirements if necessary to meet the required cost 
and schedule constraints - the driving context of this mission. 
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The LCROSS project started with a good understanding of its context, and while lean, it befitted the 
construct of the mission. How to go about actually managing to this context, however, was unclear, as 
there were no guides within the Agency documentation; we continued to leam and navigate over the 
course of the project. Full and minimum success criteria in our Level- 1 requirements provided 
outstanding requirements priorities, and therefore guidance, and that latitude was part of the 
requirements bucket. The implied requirements reserve could be consumed by dropping to minimum 
success. . .we fortunately never had to use it, but several times it was a topic of consideration. 

For a cost-capped, schedule-constrained mission, the cost and schedule buckets are critical in decision- 
making, which means the requirements bucket is the primary management trade space. However, project 
success doesn’t mean simply whittling away at the requirements every time you have a problem; you do 
have schedule and cost buckets with some resources, but when do you use them? Don’t slip into the trap 
of trying to assure success, as this drains the cost and schedule buckets quickly. 

One of the first tests of how the LCROSS team was going to behave occurred within weeks of our green- 
lighting as a project. We performed a detailed assessment of the spacecraft 1553 bus and the expected 
data traffic loading. Any given bus has a maximum amount of data that can be pushed through it in a 
given amount of time. Exceeding that level will cause bus collisions (think of a freeway) and data loss. 
The LCROSS plan was to utilize a single data bus for transferring both the spacecraft operational data 
and instrument science data. We soon discovered that the 1553 data bus could not handle the expected 
data traffic levels. We had a few options: 

1) Reduce the amount of spacecraft health and status data on the bus. 

2) Reduce the amount of instrument science data traffic on the bus. 

3) Redesign the bus (or add a second bus) to accommodate the total data bandwidth. 

Option #1 sounded good at first, but the spacecraft data loading wasn’t substantial; making reductions 
here would have been of limited value. Additionally, reducing our ability to monitor the spacecraft 
performance/health could have threatened our minimum success criteria. 

Options #2 could work because we could throttle down the frame rates of the instruments to get to the 
desired levels of traffic the bus could reliably handle. This also worked to the project context nicely as it 
made use of the requirements bucket, and sacrificed some of our full success criteria performance in 
order to assure our cost perfonnance. 

Options #3 was probably the right technical solution (and typically the default position of NASA 
missions), but was clearly the wrong programmatic choice for this cost-capped mission. The moment a 
heritage design is altered, the project is exposed to large cost risk during the development and test phase; 
this puts the minimum success criteria at risk. 

So what did we do? The clinically correct answer was Option #2. . .however we went with an Option #4. As 
you will see in a coming chapter, this mission was a capabilities-driven mission, which meant living within 
our means - in this case the hardware’s means. “Option #4” was to go back to the existing hardware design 
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and see if there was another way to relieve the bus loading problem within the existing board capability. We 
discovered an existing RS-422 capability on the board, ready to go... well nearly ready to go. We needed 
some software development to make use of it, but the bulk of the work was already done. 

Was this the right decision? Making use of the RS-422 port presented relatively little risk from a 
technical requirements point of view, but it would cost money and schedule to implement it, the very 
commodities we closely guarded on this project. Our decision to go with Option #4 was driven by a 
couple features of this particular situation: this bus-loading dilemma occurred very early in project 
development and dumping performance that early was a concern to us. Perceptions do matter. 
Additionally, we felt that dipping into the cost and schedule buckets was fairly well-bounded as we 
understood the task well. Finally, doing this work ended up solving another problem which will be 
discussed later. 

This particular decision could have been perceived at the time as starting off on the wrong foot; early in 
the project we were already acting like we didn’t understand our context, by chasing performance. We 
recognized this and were very conscientious to make sure we 

were not slipping into the trap of performance creep, thereby f"® team really Understood 

dwindling our resources. The most interesting thing was that it llOW we had to behave for 
wasn’t the stakeholders who were concerned about the this mission t(> be successful, 
decision; it was the LCROSS team. The team was so properly 

oriented with sensitivity to requirements creep, that they had some real hesitancy going down this decision 
path. It was very satisfying to see this because it revealed that the team truly understood how we had to 
behave for this mission to be successful. Similar discussions would take place many more times over the 
course of the project, and the same logic applied. This was about very deliberative risk management. 
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IVlost projects have very similar structures, even if all the details are different. This comes from the fact 
that a project is simply a collection of activities (work) that are being managed (PM) to a desired goal 
(requirements) within a set of constraints (cost and schedule) for somebody (stakeholders). It’s pretty 
universal. In this chapter I am going to illustrate some of the basics associated with the work to be 
accomplished, the stakeholders for whom we are doing the work, based on performance requirements, 
within cost and schedule constraints. In short, how LCROSS perfonned project control and reporting for 
both the project’s benefit and that of the stakeholders. This chapter is intending to simply touch on these 
topics to illustrate the kinds of controls and monitoring that LCROSS employed. 

It is important to remember that the LCROSS team was very small and so our reporting was designed to 
be sustainable by a small business office, small schedule office, small PM office, and small engineering 
office. We did have an impressive suite of data from which we could control and status the project, but it 
was no more than it needed to be. The graphics shown in this chapter are actual reporting charts used 
during the execution of the LCROSS Project. I have not changed them, figuring you’d rather see the real 
thing. Most all the charts are intentionally single-page information in order to provide me and the 
stakeholders with easily understood, one-stop-shopping of status. I could of course dig deeper, but only 
as required to address an issue. The PM needs to make sure that he defines what he needs to successfully 
manage the project, and then if the stakeholders feel they need more, negotiate with them so as to not 
create excessive reporting circumstances from the limited resources the team has available. 


The Stakeholders 

So let’s begin with the stakeholders. The stakeholders are both the person/organization who is funding the 
work, as well as others who have connection to that work, even if not formal. Because LCROSS had a 
NASA-HQ stakeholder, a program office stakeholder, a launch services provider, a co-manifested mission 
partner (LRO), and a spacecraft manufacturer, it seemed helpful to have a single diagram representing the 
various stakeholders which had some interest in LCROSS. The resultant diagram shows stakeholders and 
suppliers in a geographic format - a map. This chart (see Figure 5-1) was created by our NASA-KSC team 
members and we absorbed it with our own reporting since it gave such a great overview of all the 
connections of the LCROSS project to others, and helped define the required communication paths as well 
as potential requirements for travel. 


The Team 

Another set of stakeholders in the success of the project is of course the project team itself. As is 
traditionally the case, the standard organization chart (Org Chart - see Figure 5-2) is the common means for 
illustrating the team. Also included on the chart are the most directly-pertinent stakeholders: the program 
office and NASA-HQ program executives. These are the people your team will be interacting with most 
commonly from a reporting and statusing point of view. This chart served many purposes, including 
identifying the lines of authority, lines of responsibility, and lines of reporting for the principals on the team. 
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Figure 5-1 : LRO/LCROSS Stakeholder Landscape. 


The top name in each box is the Control Account Manager (CAM) for that area of work. I looked to the 
CAM to manage their area of work as if they were a PM for that work. They had to work within a set of 
requirements and a given budget, along a needed timeline. I chose to make the org chart mimic the Work 
Breakdown Structure (WBS) for the project so that the works areas and hierarchy were also understood. 
This simple chart was a quick reference sheet for communications and planning for the team. 


The Work 

So, as noted above this team needs to be applied to the work at hand for meeting the mission requirements. 
The “work” needs to be organized in a fashion which can reflect hierarchy and systemic organization. The 
WBS is the means for doing just that (see Figure 5-3). A WBS enables the team members to understand 
where their work fits into the big picture and to identify the providers by name. The LCROSS team 
followed the basic NASA WBS with the exception of combining the work within NASA’s WBS 9.0 
(Ground Systems) within WBS 7.0 (Mission Ops). We combined these work areas because the bulk of the 
ground systems for LCROSS were related to mission operations and were managed by a single Mission 
Operations Manager (MOM). Being willing to adjust template plans to the specific needs of the project is 
essential for the small project team. We color-coded the ownership by NAS A- ARC and Northrop- 
Grumman, as well as the other members of the LCROSS team, such as NASA-GSFC, NASA-JPL, and 
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Figure 5-2: LCROSS Organizational Chart. 

NASA-KSC. While this graphical chart made it very easy to understand the scope and organizational 
responsibility for the work, the numbering scheme was also the basis for how all the project finances and 
schedule were organized. All project efforts were funded and scheduled per this WBS structure, 
throughout the project’s life cycle. Notice the hierarchy of the work. For example, the subordinate systems 
of the spacecraft located under WBS 6.0: WBS 6.04 is where the power subsystem work was organized 
while the mechanical subsystems were located on WBS 6.07. This approach enabled “rolling-up” all the 
lower-level activities into the higher-level, CAM-managed, project-level work. 


The Plan 

Now that the work is divided into executable chunks, the order of work needs to be established. What 
pieces of work need to happen when? Network diagrams are the key to defining that work and a project 
will have very detailed diagrams which define this for ultimate use in the master schedule, but these can 
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Figure 5-3: LCROSS Work Breakdown Structure (WBS). 


be very dense diagrams and not usually very easy to use. A single-page network overview for each 
system proved to be very useful for LCROSS as it showed the big-picture. It was complete, but 
manageable. The next few figures show examples of how LCROSS illustrated its development and 
testing progress on different facets of the project. This concise, single-sheet network diagram approach 
originated with Steve Carman at NG, where they illustrated their avionics development progress as well 
as spacecraft activities in “Assembly Flow” diagrams (see Figure 5-4). The diagram illustrated the 
progress of the spacecraft build prior to going into environmental testing, Because it was so very easy to 
use, I always had a quick understanding of where the spacecraft development stood, and I could use it to 
illustrate the schedule to the stakeholders. The format we chose enabled viewing status which included 
using white boxes to show design tasks which had not started, light-blue boxes to represent work in 
which drawings had been completed, but fabrication had not yet started, and candy-cane striping 
indicated a task in which fabrication and assembly had started. Light-green outlining meant the work 
was undergoing subsystem or box-level testing and dark blue was the culminating color, indicating the 
box or subsystem was complete. Completion dates were also shown in the upper right area outside of the 
box, and red arrows leading into or out of a box meant it was on the project critical path. By inspection I 
could see the big-picture of spacecraft progress as well as details of which task currently owns the critical 
path and when it is expected to be complete. Having all this information available on a single sheet 
enabled extraction of an incredible amount of information about spacecraft status. 
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Figure 5-4: Example LCROSS “Assembly Flow” Network diagram for the spacecraft build. 


This approach worked well within the mission operations team where similar milestones and products 
were tracked with estimated completion dates and critical path identified (see Figure 5-5). Even though 
most of the products from WBS 7.0 were software and training products, this worked just the same as the 
physical hardware developments of the Ground Data System (GDS). This commonality made it easier to 
establish a common legend so that any of these network diagrams could be created and readily 
understood by anyone, including the stakeholders. It is especially interesting to note the synergies 
between the GDS development and releases working hand-in-hand with the training activities for a just- 
in-time delivery. Since the Engineering Readiness Tests (ERTs) and subsequent Operational Readiness 
Tests (ORTs) verified the Ops team’s ability to support a particular phase of operations, the GDS releases 
and operational tests would continually do a dance together in this diagram. 
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Figure 5-5: Example LCROSS Mission Operations and Ground Data System Network diagram. 


The LCROSS team also applied this network diagram approach to illustrate the path to launch readiness 
(see Figure 5-6), wherein the three simultaneous efforts which had to be accomplished before LCROSS 
was ready for launch were statused. The power of this chart came from being able to simultaneously see 
the three, distinct activities progress in advance of launch: launch review readiness, spacecraft readiness, 
and mission operations readiness. This one diagram enabled monitoring the evolving horserace amongst 
these activities: 

♦ Milestone reviews to launch (top section of the diagram) 

• Activities which are driven by the integration work of the dual-manifested payloads (LRO and 
LCROSS) - very complex and steeped in existing process, and highly visible within the Agency 

♦ Spacecraft completion and readiness to ship (middle section of the diagram) 

• Activities which are wholly controlled by the LCROSS Project and principally owned by NG as 
the builder of the spacecraft 
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♦ Mission Operations readiness (bottom section of the diagram) 

• Activities which are wholly controlled by NASA-ARC, from which the mission would be flown, 
but did have direct interaction with the NG and their Remote Operations Centers (ROCs), which 
provided backroom mission technical support 
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Figure 5-6: Example LCROSS Launch Readiness Network diagram. Notice at this point in the 

project the launch date had not yet moved out to June 18, 2009. 


Each of the Network diagrams shown was populated by the LCROSS Schedule Manager in concert with 
inputs from the appropriate team CAMs and the master schedule. I would review the finished products 
both to understand our status and to be sure I could convey our status to the stakeholders. If I saw a 
problem, I would seek the appropriate CAM and chat with him or her to determine if additional attention 
was required. These one-page network diagrams were incredibly useful and a real timesaver. 
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The Schedule 

With the team defined and the work organized, the next important step was the LCROSS project 
schedule. Since LCROSS was a secondary payload on the same ride as LRO. The LCROSS team had 
some catching-up to do since LRO had already been an executing spaceflight project in development for 
nearly 2-years at the time of LCROSS’ selection. Both projects had to make it to the launch pad together. 
The LCROSS team put together a rough schedule in the original LCROSS proposal, and upon selection, 
instantiated the Integrated Master Schedule (IMS) in Microsoft Project. This IMS was the heart of the 
LCROSS plan, but like the network diagrams which supported it, a single, high-level one page schedule 
(shown in Figure 5-7), enabled quick, high-level milestone progress reporting for the team and 
stakeholders to understand how we were doing. Our IRB chair once called this chart the “taco chips” 
chart as it represented each of the major project milestones with a triangle. The LCROSS Program Office 
agreed to use it as the basis for evaluating LCROSS ’s performance against its commitments. The 
coarseness of this single page schedule provided some latitude to make adjustments before notable 
problems required stakeholder attention, and the stakeholders understood this. The Program Office 
wanted to be sure they gave us room to remediate any problems at the project level before they became 
so big that “taco chips” were actually moving around. 
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Figure 5-7: LCROSS One-page schedule. 
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Since the LCROSS project at NASA-ARC was the holder of the IMS, we worked with NG to make sure 
we had compatible schedule tools so that their spacecraft development schedule (WBS 6.0) and other 
work areas could be imported into the project-level IMS. This was also true of the NASA-ARC payload 
team and the NASA-ARC MOS team whose CAMs all worked the schedules for their respective area 
and then the project would incorporate them into the IMS. This enabled the project manager to monitor 
all project progress. 


The Cost 

The work is now defined, and ordered, with a team ready to execute the plan. . . How much will this cost? 
The basis for the LCROSS cost came from the original proposal grass-roots and analogy estimates 
applied to the cost constrained given by the stakeholders. Time is money, so the schedule and 
procurements directly drive a resultant cost. Labor and materials costs were captured and assigned to the 
appropriate WBS, as well as cash reserves carried by the project to address unknowns that surfaced 
during the course of the mission. The result of this assignment of costs across the mission timeline is a 
LifeCycle Cost Estimate (LCCE). The Business Manager (BM) maintained a fully-developed LCCE for 
the project and formally updated it monthly with the latest changes in funds allocations, including 
reserves consumption. This LCCE was an unwieldy spreadsheet, and like the project IMS, I needed a 
simpler tool to facilitate reporting and tracking, and this is how the “Rainbow Chart” came to be. As you 
can see in Figure 5-8, the LCCE was again based on the project WBS structure and showed financial 
guidelining across fiscal years. In other words, it showed the yearly allocation of funds that the 
stakeholders needed in their planning to fund the project. The chart also divided the annual guidelines 
into labor costs versus procurement costs, where “labor” is defined as NASA civil servant labor and 
“procurements” were both contractor labor and hardware procurements. This chart has some interesting 
items. For example: Why is there no NASA civil servant labor performing Safety & Mission Assurance 
(WBS 3.0)? There were in fact government S&MA activities, but this was a function funded 
independently from the project in order to avoid conflicts of interest with the project team. As with the 
previous summary charts, the rainbow chart proved an excellent timesaver and was great for 
communicating with the stakeholders. 

The LCCE is a means for estimating what the cost of an activity will be, but you also need a means of 
tracking how costs are actually stacking-up. This is where the Cost and Obligations chart comes-in. The 
LCROSS Business Office created and updated monthly, a chart showing the ongoing cost and obligations 
performance within the broader context of the rainbow chart, but at a finer level. As Figure 5-9 
illustrates, we applied the fund phasing from the LCCE rainbow chart to a monthly timeline to create the 
Cost/Obligations tracking sheet. This chart not only to showed the burn rate of the project compared to 
the plan, but also showed the NASA guideline funding, which is the amount of funding the project has 
received to date. Since the program office doesn’t simply allocate to the project the whole project cost at 
once, this tool enables a quick inspection of how we are doing and when we may be running low on 
funding, so as to enable the project to work with the program office to assure funds arrived on a timely 
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Figure 5-8: LCROSS Life Cycle Cost Estimate (LCCE) chart (The Rainbow Chart). 


basis. Furthermore, this chart helped us understand how we were performing on a fiscal year basis and in 
particular with the fiscal year-end planning. When crossing fiscal years it is important to have sufficient 
funds going into the new fiscal year in order to cover the inevitable governmental budgetary process 
hang-ups at the start of each fiscal year. 


This chart, and others in the LCROSS set, included some typical Earned Value Management (EVM) 
metrics, although LCROSS was not required to implement formal EVM (see the top-left legend in Figure 
5-9). The LCROSS approach to monitoring project performance was to monitor schedule and cost 
variance on the IMS and LCCE. The business manager and schedule manager would monthly track the 
performance with the CAMs on their progress and then report that in comparison to the plan. I would 
receive monthly status on each the schedule and cost performance with variances tracked with traditional 
stoplight colors: green, yellow, and red. %-complete versus %-plan were compared in each of the 
product-related WBS and a variance-% revealed. This was a stakeholder- approved form of EVM 
tracking and worked beautifully on LCROSS. As variances began to appear I would go and speak with 
the corresponding CAM and discuss status and corrective action. NG did report EVM metrics for their 
work on the S/C build and other activities, and given the proportion of the S/C work (in cost), this 
information was included in the project-level reporting package as a courtesy. 

The Obligations and Cost Tracking sheet enabled me to see how the project was doing against its 
commitments, but did not address how our forward work was looking relative to where we currently are 
at any given time (i.e. how much work was left to do). This is where the Cost-To-Go (CTG) chart was 
immensely helpful (see Figure 5-10). Key to managing cost reserves was to have a good visual means for 
understanding what reserves remained and what had been consumed. The LCROSS BM came up with an 
excellent tool for recording the project’s monthly reserve position in a chart that showed our reserve 
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Figure 5-9: Obligations and Cost Tracking Sheet. 


position both on an absolute scale and on a cost-to-go basis. Whenever I presented project status to 
stakeholders, I always noted it as my favorite chart because it was so telling of project/financial health. 
From this CTG chart we could immediately see where we’d been and where we were going. The chart let 
us make judgments about when our reserve position was getting too small, thereby threatening the cost 
cap, or when our reserve position was getting rather generous and we could spend some money to buy- 
down risk somewhere. The green line in the diagram which slowly trickles downward as time passes 
showed cash reserves remaining as a fixed-fraction of the project’s Total Allocated Budget (TAB). The 
black line, which jumps-around quite a bit as you move to the right, illustrates CTG. It is important to 
understand that the CTG curve is always inclined to grow if you are not consuming reserves, because the 
denominator (cost remaining) is always shrinking as you execute work, while the numerator (reserves 
remaining) stays fixed; the ratio therefore grows. What changes the slope of the CTG is having to expend 
reserves to address a problem (numerator shrinks), or an unfunded scope change (denominator grows 
because you now have more cost liens in front of you). LCROSS was successful in working with the 
program office to assure there were no unfunded scope changes on LCROSS. The test case for this came 
when the LRO/LCROSS launch date changed due to launch manifest changes (another mission was 
“swapped” onto our launch date due to priorities), this extended the project duration for LCROSS. We 
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negotiated with the program office, that the project should receive additional funding commensurate with 
keeping the LCROSS team together since LCROSS was not responsible for the launch date change. They 
concurred and our business office set up a different set of WBS numbers just to handle the “launch swap” 
period, which we began charging to after our baselined launch date passed. We returned to the original 
WBS numbers on our actual launch date, since this resumed the originally planned activities for 
LCROSS. In this way I was able to track the LCROSS original cost performance to our original scope 
without the launch date change. 





Figure 5-10: LCROSS Life Cycle Cost-to-Go reserves tracking sheet. This chart 

incorporated the additional costs of the “Launch Swap” launch manifest date 
changes by the program office as this would affect our forward reserves planning. 
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The Procurements 

So far in this chapter we have discussed the people involved in the project as well as the work and how it 
will be organized, planned and costed. The remaining items have more to do with the implementation or 
execution of the project. One critical aspect of the project is the arrival of your piece-part procurements. 
Plans are in place for when things should arrive as your network diagrams and schedule had to make 
certain presumptions, based on promises and estimates from various suppliers and vendors. If only it 
were that easy when it comes to reality. During the implementation phase you are now taking receipt of 
these items. . .and they will not all go per the plan. In order to stay in front of issues that surface, I found 
it helpful to have what I called a Procurement Cheat Sheet (see Figure 5-11). This sheet was a single- 
page list of spacecraft hardware procurements organized by need date, and based on the project IMS. As 
with all of the tools listed in this chapter, it summarizes and makes information easier to access than 
frequent, detailed studies of the source documents like the IMS. The Schedule Manager maintained this 
chart, which included the following columns: 

♦ Item: The particular procurement item 

♦ Arrvd?: A simple checkbox to indicate with the item has already arrived 

♦ Planned Date: The date in which the item was originally planned to arrive. This date comes directly 
from the baseline IMS. 

♦ Current Date: The date in which the procurement item is currently expected to arrive. This could 
be the same date as the Planned Date, or it could be different if the items has been delayed. 

♦ Need Date: The latest date in which you need to receive the item before it enters the critical path, 
thereby driving the overall schedule. In some case, it may be OK for an item to arrive late because it 
was not near the critical path; in other cases it could be disastrous. 

♦ Slack: The number of days that particular procurement item can tolerate being late before entering 
the schedule’s critical path. 

By inspection, I could see which downstream procurements were already getting in trouble and needed 
attention. It also served as a reminder for the expected order of events per the plan. As an example, we 
didn’t need the LCROSS solar array until late into the assembly flow, as can be seen by its late need date. 
However, if we found that we were having trouble with another element, we could consider reflowing the 
solar array need date to enable later delivery of a troubled item. As another example, you can see some of 
the difficulties we had with the RAD750 processor: Our planned date was in March, but it didn’t get 
delivered until nearly three months later. This obviously was breaking the plan, and so we needed to 
adjust the schedule to accommodate the very late arrival, including consumption of schedule slack. 
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Item 
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Planned Date 
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Figure 5-1 1 : Example LCROSS single-page Procurement cheat sheet. 


The Performance 

Along with watching the procurement deliveries unfold the design is evolving and there are technical 
measures which need to be monitored. This is a monitoring role that is owned by the Project Systems 
Engineer (PSE), but watched by the PM since it is an indication of how well the design activity is going. 
Every project answers to a set of stakeholder requirements which represent the whole reason for the 
project existing in the first place. In addition to those stakeholder requirements are process requirements 
which attempt to address risk by allocating a certain amount of consumable technical margins, based on 
the design maturity. There are a number of technical measures which are good indicators for how the 
design is progressing. Derived from the project’s requirements, the Technical Performance Metrics 
(TPMs) are a way to monitor how well the project is performing to its constraints. As PM, it is important 
to understand how these matrics are doing at any point in the design activity, and as with the other charts 
in this chapter, there should be a way to conveniently do this. The PSE creates a master TPM list (see 
Figure 5-12) consisting of the critical performance measures of the mission, whether on the spacecraft, 
the payload, or the ground data system, and these are tracked over the course of the project. Some of the 
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key TPMs include total mass ( wet and dry - which means fuel-included or unfueled), power capacity of 
the system, pointing accuracy of the spacecraft for the payload, and payload performance metrics. To 
manage the margins on these measures, we compared the current predicted use (the “predict”) to the 
allocation for that technical measure. The difference between the two represented the margin available. 
We then characterized how well we were doing on that technical measure as a function of where we were 
in the project development. This approach is the equivalent of a technical Performance-To-Go, analogous 
to CTG. This chart played to the specific phases of the project and how margins are expected to be 
consumed as the design matures. (Shown in the small box at the bottom right of Figure 5-12.) 
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Figure 5-12: LCROSS Technical Performance Metrics (TPMs) sheet. 


Early in the project development it is expected to have reasonably large margins, but when design and 
integration activities were wrapping up, much smaller margins were tolerated because forward risk was 
much smaller. Analogous to CTG, if you slowly consume technical margins over time, your margin 
health actually grows given that the design is getting increasingly mature. Color-coding made inspection 
of this chart easy, with blue indicating ample margins; green indicating sufficient margins; yellow 
showing cautionary margins, and red indicating insufficient margin. How the project reacted to any of 
these conditions varied depending on the particular technical metric. In fact, one job of the PSE was to 
watch for measures of concern and potentially trade one margin with another by modifying the design. 
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The Risks 


Finally, and most importantly, throughout project execution risk must be continually assessed in order to 
understand where you sit with respect to your allocated risk classification. Recall that LCROSS was 
designated as a Class D mission, which is the most risk-tolerant classification within NASA. Our 
assessment of our composite risk, in conjunction with our cost and schedule constraints, should be 
consistent. If they are not, then action is required. A project can carry both programmatic risks and 
technical risks. There are risks which can be realized with schedule loss, cost consumption, or technical 
performance loss. The Project needs to regularly report on its assessment of its current risk position and do 
so in a concise, easy-to-understand format. LCROSS employed a standard 5x5 matrix (see Figure 5-13) to 
characterize our risks and evaluated them at least monthly with a Risk Management Board (RMB) 
wherein all the technical leads would get together for multiple hours, reassessing risks. On the 5x5 matrix, 
we would illustrate the Project’s top- 10 risks, and indicate their trending (new risk, risk trending up, or risk 
trending down). We also specified if a particular risk impacted the schedule, cost, or technical performance 
of the project. This summary sheet provided an excellent holistic overview of the project’s risk position 
which would enable me to see where the top risks lie, and also report them out to the stakeholders. 
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Figure 5-13: LCROSS Risk Management tracking chart. 
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This overview of the Project Basics was not intended to be an exhaustive discussion of all the metrics and 
means for monitoring a project. There are certainly more and less detailed ways of accomplishing the same, 
but this does represent the LCROSS approach to both managing and reporting status for both internal and 
external use. I found this approach to be very efficient for . 

meeting most everyone’s needs. Every stakeholder community 

will be different and every PM will have different levels of do should bf helping 
depth required to manage a project, but our LCROSS approach move the project forw aril 

flows from the ideal that every piece of work you do should be you wasting resources, 
helping move the project forward or you are wasting resources. 

That ideal is difficult to fully achieve, but it was my target in setting-up the LCROSS reporting. Every 
activity spent on non-valuable work is displacing some other activity, and it is important that reporting not 
take on a life of its own. Lrequently, reporting becomes something which must be fed and the reporting is 
not allowed to focus on what is needed to enable the project. 
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Every project has a start and an end wherein the purpose of the project is (hopefully) realized. It’s all 
that in-between stuff, however, that defines if it will all work-out as hoped. Within NASA there are well- 
prescribed project phases which define the steps NASA projects go through as they move from start to 
finish. These phases and the corresponding “gate reviews” have evolved over many years to represent the 
favorable way to get to success. The problem of course is that there are many different sizes of projects 
with differing customers and risk positions which make a since pathway unoptimized. At the same time, 
it does make sense from a corporate point of view to have “a way” that all projects follow so that the 
people of that organization can speak in common terms; so that people and teams have common 
understanding of relative completion and status. LCROSS therefore worked within the standard project 
framework for NASA missions and chose to tailor the detailing therein to suit the realities and context of 
the LCROSS project. 
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Figure 5 2 The NASA Project Life Cycle 
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Figure 6-1 : LCROSS Review schedule. 


LCROSS worked within the NASA project lifecycle which illustrates all phases in a project from 
concept to completion 1 . The project lifecycle splits the effort into two halves: Fonnulation and 
Implementation. As the name suggests, the Formulation phase takes the project from the genn of an idea 
through preliminary design. Upon acceptance of a preliminary design, the project moves into the 
Implementation phase, in which the design is completed, and the system is built, tested, launched and 
operated. These project phases include sub-activities which comprise everything from the initial concept 
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studies through initial and final design, assembly, testing, launch, operations and close-out. The LCROSS 
project experienced all these phases, following the “Robotic Mission Project Reviews” line in Figure 6-1 
above. The annotations below the chart show LCROSS ’s actual schedule performance, including a 
couple added reviews, which made good sense for the project. Given the small size of the LCROSS team, 
the short time we had, and the limits on our budget, we had to move through the reviews with alacrity. 
The LCROSS pace of development and reviews was extraordinary for missions of this size. Below I 
describe the typical project phases in more detail and explain particular LCROSS approaches. 

Project Formulation: Requirements Definition 

This phase is where the project is initially formulated, where the fundamental purposes and objectives of 
the mission are defined. These objectives frequently originate from needs, goals, and objectives (NGOs) 
discussions with the stakeholders, and are evolved during mission scoping. All subsequent system 
requirements are derived from these objectives, since they are the whole purpose of the mission in the 
first place. While we are used to hearing that the requirements should not be changed over the course of 
the project (and ideally they don’t), in reality it is the 
objectives which should not change. Requirements are 
merely the flow-down from the objectives, with the “Level- 
1” requirements being the ones which the Agency will use to 
detennine your project’s success. Level- 1 requirements, by 
their high-level stakeholder ownership, take-on the 
importance of “objectives” and should not change. For 
LCROSS, the key objectives were to design, build and 
launch a spacecraft comanifested on the LRO launch vehicle, which would send an impactor into a 
permanently-shadowed crater on the moon, enabling detection of the presence of water, for no more than 
$79M. These objectives are normally first evaluated at a Mission Concept Review (MCR), but for 
LCROSS, our MCR was supplanted by the original mission proposal and our later project selection was 
the equivalent of passing that review. 

The LCROSS project had considered a large number of impactor options, from carrying heavy spheres to 
be dropped in multiple locations to improve lunar surface understanding, to mini-impacting sensor 
probes shot into various places on the moon. However, when the idea came up of using the upper stage of 
the launch vehicle as the impactor of the mission, we knew this would be very attractive. We generically 
called the launch vehicle’s upper stage the EDUS (Earth Departure Upper Stage) since we didn’t know 
which launch vehicle we would fly on at that time. The most attractive part of our mission plan was that 
we got the 2300-kg EDUS impactor for free, as it was not counted against the mass budget of the 
LCROSS spacecraft. This mission concept, coupled with the mass, schedule, and cost constraints of 
LCROSS, became the basis for the mission. 


Level- 1 requirements, by 
their high-level stakeholder 
ownership, take-on the 
importance of ‘‘objectives'" 
and should not change. 
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The concept definition phase was therefore the proposal phase for LCROSS. We took the key objectives 
and turned them into a draft staffing plan, created our proposed schedule, budget and WBS, and 
developed a technical approach for accomplishing the mission, all culminating in the LCROSS proposal. 

With the key objectives in place and the basic project approach approved at project selection, it was time 
to flow down requirements to increasing levels of detail in order to enable building the system. Since we 
conceived LCROSS at a project-level set of objectives (Level-3 or L3), the superior, abstracted 
objectives (LI, L2) had to be generated. NASA Headquarters owned the stakeholder LI requirements, 
which came to be the ultimate key objectives; the L2 requirements were held by the Program Office, and 
were more detailed than the Lis, but less detailed than the L3s. The finished product was a nice waterfall 
flow of LI requirements (objectives), down into the L2 requirements, down to the project-level L3 
requirements. L4 requirements and below were flowed down from the L3s by each of the project Control 
Account Mangers (CAMs). The goal was to subdivide the work into elements that would ultimately 
support the project L3s. The L4s applied to all suppliers of products to the project: 

♦ NG owned the L4s for the spacecraft 

♦ NASA- ARC owned the L4s for the payload instruments development, the Mission Operations 
Control Room (MOCR) and Ground Data Systems (GDS) 

♦ NASA-KSC owned the L4s for the launch vehicle needs to support LCROSS (and LRO) 

Because the L3s are parent to all flowed-down L4s, L5s, etc., it was important that the L4 providers 
participate as part of the project team. This would get buy-in and assure that the L3s were achievable 
based on the provider’s experience. The project left the detailing of the L4s, L5s, etc., to the providers. 
We may have been interested in witnessing the flow-down, but the provider remained the owner of these 
flowed-down requirements. It was essential for the providers to develop these requirements so that they 
would be in alignment with their internal organizational needs and the process appropriate for the 
project’s risk position. Across the board, LCROSS was based on leveraging the existing partner’s 
capabilities and letting them do what they do best. 

The requirements development activity ends with a System Requirements Review (SRR). This review is 
attended by the stakeholder community, and its purpose is to be sure the project-formulated requirements 
are appropriate for the key objectives from which the mission was selected. The reviewers should also look 
to see if the lower-level requirements are design-achievable. That-is, do the independent reviewers believe 
that the requirements which have evolved from the objectives are feasible for the intended design concept. 
As noted earlier, these requirements, which originated from the key objectives, can evolve over time as 
necessary. Since this is one of the earliest reviews, this is also a good time to work with the stakeholders to 
assure mutual understanding of the nature of the reviews and expectations therein. This is a topic of much 
discussion later as LCROSS was a small Class D mission. Resource limitations and risk position are good 
topics to shape the nature and duration of the reviews and this is a good topic to address early. 
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Project Formulation: Preliminary Design 

The Preliminary Design phase is where the requirements generated in the previous activity begin to get 
realized in initial designs. This phase is important for assessing the feasibility of meeting the project 
requirements. Up to this point, the planning has been based on estimates for what would be available and 
achievable. This phase takes stock of those earlier estimations and attempts to make them real by looking 
at the actual designs and materials needed to implement. While this implies a line in the sand for the 
maturity of all the designs, in reality many of the sub-system designs may be beyond the PDR level of 
maturity. That is, some of the designs are nearly completed to the critical design level. This is fine, since it 
increases the confidence that the proposed implementation will work. However, it is also important that 
some design elements not get too far ahead of others. If an element in the preliminary design breaks, other 
more mature design elements may need to be reworked, thereby wasting expensive design work already 
completed. The project relies heavily on the Systems Engineering team to watch for this happening. The 
SE team needs to be looking for the key elements in the design, assuring everything is holding together, 
despite the different maturity levels. Interface Control Documents (ICDs) help with this assurance, as draft 
versions should be getting agreed-to and written-down during preliminary design. Technical Performance 
Measures (TPMs) margins assessments are also being drafted to help defend the design and realism in 
achieving the requirements. The big three TPMs - mass, power, and volume - need to have defensible 
stories on how they fulfill the objectives and meet the stakeholder-stipulated risk position. 

It is tempting to spend a lot of time in this phase trying to get everything just right, but this becomes cost- 
prohibitive. A lot of expensive, salaried individuals are working the design, and moving through this phase 
quickly helps contain cost. Since LCROSS was not developing new technologies (by intention), the 
maturity of the systems design was quite high. Nevertheless, we still received comments from some in the 
stakeholder community to do this or that beyond the original plans, and arguably beyond expectations of 
this preliminary design phase. We had to actively manage the stakeholders during this time to avoid 
requirements creep or review creep (the slow accumulation of “minor” changes or requests which build up 
to a significant burden). Minimizing the degree of formality enabled us to keep moving forward.. 

This design phase culminated in a Preliminary Design Review (PDR). The LCROSS project led the 
review, but it was hosted at the contractor site since the bulk of the PDR effort was spacecraft-related. The 
NASA-ARC payload design was presented, as well as a deliverable to spacecraft integration down the 
line. The LCROSS Mission Operations System (MOS) team also presented material to show that their 
planning was tracking the design of the spacecraft and any mission operations-related constraints coming 
from the spacecraft development. They later held their own MOS-PDR, since mission operational 
maturity tends to lag spacecraft design maturity. The launch vehicle provider was also at the PDR and 
even presented material to assure no plans were taking hold that would compromise later launch vehicle 
integration or compatibility. With LCROSS, this was an especially sensitive topic, as the selection of the 
launch vehicle was not made until LCROSS had already headed into design. This meant there were 
differing perfonnance capabilities and constraints affecting the designs and margins for LCROSS. 


45 


Chapter 6 — Managing the Project Phases 


Where required, Requests For Action (RFAs) may be written to capture some of the uncertainty and risks 
associated with elements like the launch vehicle. The PM and SE should be watching the preliminary 
design to be certain the key objectives are still being met against the backdrop of the requirements from 
SRR. Are any requirements changes requested? Why and what does this mean for the project objectives? 
If this is a cost-capped mission, it may even make sense to revisit requirements if they are driving the 
design to an expensive result. It is essential to remember those driving objectives and if one of them is to 
fit within a cost-cap, what other requirements must be shed to enable the planning to close? 

The preliminary design becomes the technical basis for the subsequent Integrated Baseline Review / 
Confirmation Review (IBR/CR). This pair of reviews is the Agency’s checkpoint for assessing the 
baseline schedule and cost viability of the technical plan. This is where the Agency can cut bait and 
cancel the project if the project context is looking unrealistic, or riskier than it is supposed to be. 
Preparation for this review is less about the technical design and more about the programmatic feasibility. 
The Business Office team will be working very hard in advance of this review to make sure that all the 
financial planning is in place to not only close on the planning story, but to show how formal business 
reporting will occur going forward. Upon a successful IBR/CR, the project is now officially tracking its 
commitments like schedule and cost. 


Project Implementation: Critical Design 


There is a natural tension 
between spending a lot of 
time in this phase getting 
everything to high eonfldenee 
versus the large cost of doing so. 


The Critical Design phase is the first phase of project 
Implementation (no longer Fonnulation). Entering this 
design phase means you have secured stakeholder 
endorsement of your plans and your preliminary design. 

The Implementation phase executes the previously- 
approved preliminary design. Like the previous phase, a 
number of salaried, highly-skilled designers work all the 
details of the mission design (spacecraft, instruments, mission operations, etc.) and so there is a natural 
tension between spending a lot of time in this phase getting everything to high confidence versus the large 
cost of doing so. The SEs need to continue to assure the design is remaining plausibly stitched together 
through completion of the ICDs. The importance of maturing the ICDs cannot be overstated. So much of 
the system capability and liens/threats against the TPMs are defined by the agreements captured in the 
ICDs. Late definition can be disastrous. LCROSS struggled with the late definition of the launch vehicle 
ICD which put a very real threat against the LCROSS system design. We were able to work through it, but 
we were continually uncertain as to some technical constraints as we were moving through this activity. In 
addition, there may be some fabrication and procurements during this phase, even though this is supposed 
to be exclusively a design phase. In reality projects frequently have long- lead procurements which need to 
get a head start so as to not threaten the IMS critical path. The same is true for some fabrication activities. 
It is essential to pay extra attention to these elements and their interfaces since you are committing to them 
early in the design and are likely to be stuck with them downstream if there is a problem. These 
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accelerated items may be very well understood and present little technical risk to the project, or they could 
be a driving element of the overall design. Appropriate contingencies must be in place should something 
not go as planned with the long-lead procurements and builds. 

This phase culminated in a Critical Design Review (CDR). The LCROSS project led the review, but was 
again hosted by NG at their site as the bulk of the CDR effort was still spacecraft-related. The NASA- 
ARC payload had to be at a similar level of maturity in order to factor into the forthcoming spacecraft I&T 
flow via the now established ICDs. Additionally, the MOS crew needed a good picture of the operational 
environment of the spacecraft, including operational constraints, which evolved in the design. 

Prior to the CDR it is important to hold a number of peer-level reviews in the relevant WBS areas which 
will then be highlighted at the project-level CDR. A single CDR event, no matter how many days in 
duration cannot accomplish the level of review of a detailed tabletop session with experts in the room, 
focused on a particular subsystem (examples being the propulsion system, or the mission payload). These 
tabletop peer reviews should reveal design weaknesses and draw on the expertise of the reviewers to 
improve the design (within constraints). Then at the CDR the details of the review do not need to be 
rehashed, but simply summarized for a holistic snapshot. It is a good idea to have one of the standing 
review board members attend the peer review so they can act as an endorser of the peer review at the 
CDR-level review. 

There can be discussions during this phase about trading between Ops risk and spacecraft design risk. The PM 
and SE should be evaluating the design to confirm that the key objectives are still being met. It is possible for 
some design items not to be complete at CDR, but they must either have very simple requirements to satisfy or 
have sufficient contingency options associated with them to enable passing the CDR. 

Project Implementation: Fabrication, Integration and Test 

So now the design is done and it is time to build what you’ve designed. As noted in the previous section, 
some fabrication may have already begun, as there really is no clean line when the fabrication phase 
begins; you can’t wait for every last design item to be ready or you will have poor schedule performance. 
This is one of the most challenging phases from the standpoint of schedule. In earlier design phases you 
generally have better control over the activities and constraints of the team. In this phase, the project 
frequently depends on the performance of acquired parts and subsystem providers as well as realities 
associated with available craft skills and facilities. Competing projects can represent threats to your 
staffing needs. Your team has become very familiar with your system design and is invaluable to your 
project success, because of that familiarity, so losing staff at this time should be watched closely. 
Sometimes staffing changes come by way of unintended stakeholder decisioning; be sure to elevate 
staffing impacts with the stakeholders immediately so that they can understand the consequences. They 
may choose to proceed per their original plan, but now you’ve officially noted the impact to your project, 
providing potential for recourse, depending on the degree of impact. The forward plan may have to be 
adapted in this phase to accommodate these planning threats. 
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The delineation between Fabrication and Integration & Test is frequently blurred due to the complexities 
of delivery and assembly. Systems engineering needs to pay close attention to how the manufacturing of 
key elements is going. Issues may be as simple as delayed arrival of manufactured parts (requiring 
schedule adjustment), to manufactured parts not being able to achieve the design (entailing adjustments 
to the ICD and other designed elements). They key is to kn ow of these issues as soon as possible - in this 
phase and not in the next one. You may even discover schedule opportunities with early arrival of 
acquisitions which should be replanned into the schedule where possible. 

Use your executive management to influence vendor delivery dates if you have the ability. Sometimes a 
vendor simply needs to understand the relative importance to them to make a particularly important 
delivery date. This is a good place to apply incentives if you can, which may not just be financial. 
Understand what the vendor’s motivations are and craft a plan to help them with their needs and associate 
it with the delivery of your elements. Meanwhile, if that doesn’t work the way you need, have plans for 
late deliveries, including changing the order of integration and test activities to play to your realities. 

Because this phase is based on the realities of manufacturing, it’s likely that you will have to adjust some 
requirements. Most of these will be low-level (L6, L5) changes, but could potentially propagate to L3 
project level. The PM and SE must be alert to requirements changes that propagate to key objectives risk. 
It is not a bad thing to adjust requirements at the lower levels if they help to secure the key objectives in 
a timely, cost-effective manner. 


... this phase is about learning what 
the spacecraft is and what it is not; 
understanding how it behaves 
under normal operations and 
various forms of stress. 


The I&T phase is really an extension of the 
previous fabrication activities; here the piece-parts 
just made are assembled and tested. This is the 
beginning of the verification process, as you’ve 
designed the spacecraft, built or acquired the parts, 
put them together, and now are testing the system 
to see if you got what you intended. You are now 

going to see what you ’ve built. You want to make sure the key objectives are still being met. Since you are 
working against cost and schedule constraints, the goal is not to force the machine to meet the designed 
requirements, but to fully characterize the machine. In the I&T phase you make sure that the spacecraft is 
capable of meeting the key objectives, which includes the performance of the instruments, and the ability 
of the mission operations team to fly it. Beyond that, this phase is about learning what the spacecraft is and 
what it is not; understanding how it behaves under normal operations and various forms of stress. In this 
phase you are becoming smarter about what you will later fly, and all the notes, observations, and 
measurements you take in this phase will be invaluable when you are flying the mission and an anomaly 
occurs. When it does, you’d like to be confident that in as much as you can afford it (programmatic 
constraints), you know the personality of the spacecraft so its behavior on orbit is decipherable. 
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Figure 6-2: NASA Lunar Prospector mission lunar water detection data at the NASA-KSC 

Visitor's Center, years earlier making the case for why a LCROSS mission was necessary. 


The stakeholders and the project should have had a discussion about an incompressible test list (ITL), if 
there is one. This is the list of absolute minimum testing that the project will perform on the spacecraft no 
matter what the circumstance. In the case of LCROSS we had our test plan fully defined, but successfully 
argued that a cost-capped, schedule-constrained Class D mission should not have an ITL since the residual 
technical risk associated with testing needs to be balanced with cost and schedule which are equally strong 
drivers. Each project will be different and the key is to make sure the stakeholders fully understand the 
project approach. 

Testing must be quick and efficient. It cannot be so extensive as to establish reliability, but needs to be 
sufficient to show mission-worthiness. The NAS A- ARC developed payload suite made considerable use 
of commercial instruments, which means LCROSS testing of those elements was specifically designed to 
check the space worthiness of these commercial products. On the other hand, instruments which had 
already been certified and demonstrated in space applications and met their vendor-specified 
performance needed little if any subsystem level testing, deferring to the spacecraft-level testing, without 
substantial risk growth. 
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In general, LCROSS employed acceptance level testing for most systems elements, only invoking and 
protoqual level testing for select items. Acceptance level testing observes the actual flight hardware’s 
ability to perform in the anticipated environmental conditions (temperature, pressure, vibration, EMI, 
etc.), while protoqual (proto-qualification) testing challenges the hardware to environments that are 
beyond the worst-case expected conditions for the mission. In a Class D framework, acceptance-level 
testing is sufficient for environmental verification and can save considerable labor and facilities resources. 

The first type of testing typical for spacecraft enviromnental test is vibrational testing. This testing is 
fundamentally checking for workmanship issues during assembly and helping to assure that the spacecraft 
will survive the harsh environment of launch. I found it especially interesting to come to understand that 
much of the durability in the design of a spacecraft is merely to survive launch and not driven by the actual 
on-orbit needs! The vibrational testing can come either by way of either a shaker platform or an acoustic 
chamber by which pure acoustic energy vibrates the spacecraft at predefined frequencies, amplitudes and 
durations. LCROSS employed acoustic testing in a large acoustic facility at NG. 

The second but arguably most important type of spacecraft testing takes place in a Thermal Vacuum 
Chamber (TVAC), where the spacecraft is exposed to repetitive thermal cycling, alternately heated and 
cooled inside a vacuum chamber to simulate the environment of space. Space is very cold, so the purpose 
of the cold-side testing is self-evident; however, spacecraft will also find themselves getting very hot 
depending on their orbital design and proximity/orientation to the sun. LCROSS employed only 1.5 
TVAC cycles, which is low by typical spacecraft testing standards. Some spacecraft see nearly a dozen 
thermal cycles to convince the spacecraft is worthy of the mission. The question once again comes down 
to risk. Lor LCROSS, we felt that most failures revealed during TVAC testing appear in the first cycle or 
two and TVAC facilities and support personal are very expensive; so given our risk and cost positions, 
we completed the testing after 1.5 cycles. Given that the LCROSS team had to be responsive to cost and 
schedule constraints, when additional people were needed for testing like TVAC, technicians were briefly 
added from larger projects. 

Out TVAC testing began by raising the temp to liberate any accumulated volatiles on the spacecraft before 
doing anything else. We then followed with a cool-down to cold temperatures to simulate the launch 
vehicle’s ascent into space after launch. This part of the cycling included a cold soak, wherein the spacecraft 
simply sits in a very cold, representative temperature. The spacecraft is even operated at this temperature 
being the best, terrestrial confidence builder that the spacecraft will work on orbit. The temp is again raised 
back to high temps and sits at a hot soak survival temp. Spacecraft operations are also perfonned here for 
the same reasons as during the cold soak. Finally, the temps are returned to back to ambient temperature, 
concluding the 1.5 cycles test. The purpose of this testing was not to test to failure, but to simply check for 
workmanship-related issues, which frequently appear quickly with thermal cycling. 

Writing and executing the test procedure are critically important to seeing that the spacecraft’s behavior is 
well characterized, but within the framework of Test-As-You-Fly (TAYF). TAYF is a philosophical position 
wherein you attempt to test the spacecraft and operational crew in the same way that the mission will be 
operated... and nothing else. The idea is that a cost-capped Class D mission cannot afford to run through 
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every possible failure scenario in test and training - there are not enough resources. Therefore, you test in 
the way and under the conditions that you expect the spacecraft and the operations team to operate. 


Small accountable team dynamic 
with badgeless technical insight 
versus standing army oversight. 


LCROSS had a relatively small team performing I&T 
at NG, with participation by the NASA-ARC liaisons. 

These liaisons were present specifically to learn about 
the spacecraft and its operation and to provide 
communications back to the project leadership at 

NASA-ARC. These liaisons would later become our flight controllers. The I&T team at NG embraced 
the NASA liaisons, joining a relatively small team consisting of a single Subject Matter Expert (SME) in 
each of the relevant engineering disciplines, plus technicians and sometimes new folks coming in to do 
training. Much like the broader LCROSS project team, a small I&T team meant communications lines 
were clean, activities scheduling were simpler, and overall cost burden was lowered. As noted by the Air 
Force Research Lab (AFRL 2 ), you want a “small accountable team dynamic with badgeless technical 
insight versus standing army oversight, a preference toward prototyping/test over mounds of paper, and 
essential documentation only - who will really read it?” 


LCROSS’s I&T program began with the spacecraft avionics in what was called a “FlatSat” (as in Flat 
Satellite) performed well in advance of spacecraft integration. Its purpose was to assemble the actual flight 
avionics hardware (laying on tables and interconnected with wires and simulators), running the actual flight 
software, to see how well they ran together and then address any issues right then before having everything 
integrated on the spacecraft, where it would be much more difficult and expensive to remediate. This was a 
very productive activity for LCROSS, as we did indeed discover unexpected interactions between some of 
the hardware. 


The problem is when to bring this FlatSat activity to a close. ... If you cut the activity short, you are merely 
pushing any residual (perhaps undiscovered) problems later in the schedule where remediation will be 
much more difficult. At the same time, if you drag the FlatSat out, the downstream activities will be 
delayed, hemorrhaging both schedule and cost as you pay for some staff to wait for FlatSat activities to 
come to a close (the standing army problem). Furthering the schedule problem, you have scheduled time in 
various test facilities (e.g., TVAC or acoustic chambers) as part of the project’s I&T flow, and you may lose 
your scheduled test window due to the facilities’ high demand and deep waiting list. 

During I&T testing, when discrepancies crop up in testing against the requirements, Discrepancy Reports 
(DRs), or Problem Failure Reports (PFRs) are created to carefully note non-conformances. Not all DRs 
and PFRs need to be remediated to support the original requirement; a DR could be accepted or waived if 
it does not involve a critical performance parameter affecting the key objectives. In fact, it may be that 
this late in the project flow, remediation to meet a requirement could threaten a key objective, in the form 
of schedule loss, cost reserves hit, or even impact to the launch window. In cases of a non-conformance, 
it is important to involve the mission operations team, because there may be an operational work-around 
which can resolve the issue. Even if not, they need to be aware of any changes (via the DR/PFR) so that 
operations constraints can be revisited. Remember, everything learned about the spacecraft during this 
time is directly relevant to how it will behave on orbit and the operations crew needs to understand that. 
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The LCROSS project took this mission operations tie with I&T to an additional level. By choosing the same 
command software and system that was used for I&T as used for in-flight activities, the I&T test ground 
system software was ported over to the operations environment so that the software used to check out the 
spacecraft functionality on the ground was the same as was used to fly the spacecraft on its journey to the 
moon. In fact, we even ported screens that were developed to support I&T testing and used them to fly the 
spacecraft. This risk-reducing idea enabled the operational environment to be verified through the I&T 
testing process. At the same time, people sitting on console during the actual mission would be getting 
familiar during I&T with the ground system and how the spacecraft worked. In fact, the personality of the 
spacecraft was being witnessed through the ground system screens and software, becoming even more 
familiar for the operations crew during the mission. It should be no wonder that the NASA-ARC liaisons 
became key members of the operations team, and that many of the NG I&T team were put on console at the 
NG Remote Operations Centers (ROCs) to similarly bring that corporate knowledge to the mission. Before 
moving into the integration phase, a Systems Integration Review (SIR) is held. 

Project Implementation: Launch Vehicle Accommodation 

LCROSS launched out of Cape Canaveral in Florida as part of the NASA Launch Services Program 
(LSP). The LRO and LCROSS payloads were launched atop a United Launch Alliance (ULA) Atlas-V 
launch vehicle using stacking/integrating procedures and plans defined by the payload integrator 
(Astrotech) and the ULA Atlas teams. At this phase, the role of the project was to continue assuring its 
requirements were met, including any special needs associated with the spacecraft, the GDS, and any 
final testing of the spacecraft before launch. 

Shipping the spacecraft is something to consider as an opportunity. It is typical for spacecraft to be 
shipped by air from the integration facility to the launch vehicle; however, the preparations and cost 
involved in this could be reduced if shipping by ground is a possibility. LCROSS shipped by ground and 
saved a considerable amount of money. If you can afford a slightly slower delivery to the launch site this 
may be a sensible option. 

The LCROSS mission was planned around a Ship and Shoot approach wherein the spacecraft would be 
fully checked out prior to shipping to the Cape and then upon arrival would be essentially ready to shoot 
(launch) after some post-shipping verification of functionality, charging the batteries and filling up the 
tank. We did decide, however, to conduct a final End-to-End (E2E) test at the integration facility. This 
test consisted of commanding the spacecraft in the same way it would be commanded when on orbit, 
through the same hardware. We set up a secure internet connection between Astrotech in Titusville, FL to 
NASA-ARC at Moffett Field, CA, where the LCROSS Mission Ops Control Room (MOCR) was 
located. We employed a mobile Deep Space Network (DSN) trailer connection to radiate commands and 
telemetry from the LCROSS spacecraft transponder through antennae hats (which capture the radiated 
energy from the spacecraft transponder), back through the internet to the NASA-ARC MOCR. It was an 
incredibly thorough test which greatly enhanced the project risk position. This test was arguably not 
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required for a Class D mission, but we factored it into the flow because we had the time to do so at the 
Cape due to launch manifesting delays beyond our control. 

Remember, while a risk tolerant designation means 
you can accept more risk, you should always be 
looking for opportunities, (within your means), to 
lower your risk position. This E2E test was judged 
to be worthwhile both for the ultimate verification 
it represented, and because we were unable to 
conduct this testing back at NG’s facilities prior to 
shipping. Because there was some risk introduced by conducting this test (potential misconfiguration of 
ground equipment harming the spacecraft), this test was not conducted without some trepidation. Each of 
these decisions carried risk trades to consider. 


While a risk tolerant designation 
means you can accept more risk, 
you should always be looking for 
opportunities, (within your means), 
to lower your risk position. 


Project Implementation: Mission Operations 

In the I&T phase, we learned of the spacecraft’s personality under many different situations. On-orbit 
mission operations is where that learning is put to the test. Not unlike a medical doctor asking for a baseline 
EKG in order to characterize a patient’s heart while healthy, the I&T testing results from before launch 
enabled us to see if spacecraft behavior was altered since launch. Did the launch environment change 
anything? Loosen anything? Break anything? Were thruster alignments holding, or did we have a new 
calibration basis? 


Once the on-orbit spacecraft communicates with 
the mission operations team, the team immediately 
begins assessing how the machine is now doing. 
The launch environment is frequently the harshest 
environment the spacecraft will ever see - how did 
it do? You now witness the actual performance. 
You hope it is close, if not identical, to the 
extensive testing and characterizing you did prior 
to launch, but even if it changed, you have what 
you have. This is the vehicle you have to fly. 


You now witness the actual 
performance from the machine. 

You hope it is close, if not identical, 
to the extensive testing and 
characterizing you did prior to 
launch, but even if it changed, 
you have what you have. 

This is the vehicle you have to fly. 


On LCROSS I depended on my PSE to be the ultimate authority on all matters technical, but in the case 
of on-orbit performance, it may actually be the I&T lead who has the most relevant knowledge of the 
spacecraft and its operation; the PSE should work closely with the I&T lead and associated crew to help 
make technical assessments of the spacecraft performance and health. Since LCROSS had a very small 
team, the knowledge of the spacecraft’s performance was shared across multiple disciplines and people 
and we tried to make sure they were part of the operations team, or at least available. This principally 
meant the Subject Matter Experts (SMEs) from NG, but also included our NASA liaisons, and the NASA 
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SE team. It was important to get people who were directly involved in I&T to be members of the flight 
team. The liaisons qualified for this role, as did a number of NG staff who were monitoring spacecraft 
activities from the Remote Operations Center (ROC) that LCROSS established at NG. The ROCs were 
established to actually save money for the project without compromising responsiveness while on orbit. 
Instead of paying NGAS personnel to come up and reside at NASA- ARC with all the travel costs therein, 
they lived in their own homes and worked schedule shifts remotely. With the appropriate voice loops and 
video for streaming communications, we could make the operations environment nearly seamless as 
compared to everyone being in the same room. 

The flight team consists of the crew that has been training for many months earlier on the ground system 
throughout spacecraft testing. The ideal scenario is for “designers [to] become testers, [who] become 
operators .” 3 


In this mission operations phase you reap 
the fruits of your labor. The mission exists 
for the key objectives and you are 
about to realize them. 


In this mission operations phase you reap 
the fruits of your labor. The mission exists 
for the key objectives and you are about to 
realize them. At this phase of the project 
the focus is all about the success criteria. 

Issues on orbit will force you to make 

considerations with these success criteria in mind. Opportunities may appear and you have to evaluate 
whether they are worthy of pursuit or would represent risk to the key objectives. The LCROSS key 
objectives were to fly a mission to the moon, to impact it, and to measure what was lifted. Our Minimum 
Mission Success criteria had to be assured, but we ideally would be able to achieve Full Mission Success. 
Before moving into this mission operations phase an Operational Readiness Review (ORR) is held. 


Project Implementation: Decommissioning/Closure 

LCROSS did not have a fonnal decommissioning phase, since the impact represented the ultimate 
“decommissioning.” But we did hold a Critical Events Readiness Review (CERR) with the stakeholders to 
confirm that we were properly prepared for this final important event. We discussed the training and 
rehearsing we had completed in preparation for the impact, and summarized the spacecraft performance for 
everyone to make sure we understood its condition going into the final event. 

Clearly LCROSS did not have the look of a 

typical mission in this phase. Most missions hold Most missions hold decommissioning 
decommissioning reviews at the end of their reviews at the end of their USCtlll life; 
useful life; LCROSS used the impact to end its LC ROSS used the impact 

useful life! A more typical mission would enter eild its Useful life’ 

into its decommissioning review following a 

successful primary mission. A decision of how to bring the spacecraft’s mission to a close is discussed, 
based on circumstances of utility, available propellant, available funding, and any other aspects of the 
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spacecraft state compared to the original plan. Should the mission impact it into a nearby body (a la the 
Lunar Prospector mission, which was repurposed as a lunar impactor after its orbital life was complete)? 
Should the mission simply park the spacecraft in a graveyard orbit for disposal or future reuse? Is there a 
new purpose for the spacecraft? Should the remaining fuel be dumped? What final commands are needed 
to execute the plans? There are many considerations when a mission comes to a close and each mission 
will carry its own particular opportunities. 

The LCROSS experience, while a very different kind of mission, followed the same project phases as 
any NASA mission. The key is to tailor the efforts in each phase to be appropriate with the risk 
classification and programmatic constraints of your project, which requires a fair amount of interaction 
with the stakeholders to make sure they are being “brought-along” with the thinking.. In a cost-capped 
mission, decide very carefully how long you should spend in each phase - the ultimate risk trade-off: If 
you spend ample time in test your technical risk recedes, but your programmatic risk skyrockets. If you 
leave testing early your cost and schedule are contained, but your technical risk may soar. On a risk- 
tolerant, cost-constrained mission, the key is to do your best to strike a balance between the two. 


End Notes 

1. http://nodis3.gsfc.nasa.gov/npg_img/N_PR_7123_001A_/N_PR_7123_001A_.pdf 

2. “AXIOM Space Experiement Brienfmg for Industry,” Beal, B., Hausgen, P., AFRL, Mar 8-9, 2011. 

3. NASA FBC TASK FINAL REPORT - Rules of Engagement for FBC projects,” Spear, T„ March 2000. 
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Chapter 7 — Managing the Project Staffing 


Once LCROSS was given the green light, the weight of what we won sat with us - one of those “WE 
WONL.yi&e.s, we won..." kind of moments. In order to help me keep my perspective on what just 
occurred and what was about to happen, the chair of the LCROSS Independent Review Board (IRB) 
jokingly told me to recite, “It will be fun. . .1 will enjoy. (Repeat).” We had to get moving. 

Project staffing is always a tough issue for cost-constrained missions. Limiting staff size saves money, 
keeps the team efficient in decision-making, and tightens communications. But too small a team puts a 
substantial burden on the individuals and increases the risk that mistakes will be made or that people will 
burn out. There is no single approach to this so you need to be adaptable, making augmentations or 
deletions as needed. 

The LCROSS core project team consisted of the following: 

♦ Project Manager (and a deputy) 

♦ Project Systems Engineer (and a deputy) 

♦ Schedule Manager 

♦ Project Control Manager (CM/DM/Reviews coordinator) 

♦ Business Manager (plus a Financial Analyst for the early part of the project) 

♦ Contracting Officer (75% time on average, but will be very demanding in the early phases of the 
project) 

♦ Administrative Assistant (50% time) 

♦ Various Control Account Managers (CAMs) (Payloads/Science, Mission Operations, Launch 
Systems) 

This team was very effective in general, but there were some trouble spots where I would augment the 
team in subsequent projects: 

♦ Deputy Business Manager: At the beginning of the project we made extraordinary efforts to 
define the project costs, organize the financial products to support the LCCE, prepare for monthly 
financial reporting, and get ready for the project’s Confirmation Review (including the handling of 
independent cost estimates (ICEs)). Having a deputy BM who can fill the roles of both the Business 
Manager and the financial analyst assures bench depth and capacity for the early-project activities. 
Later on, LCROSS was able to successful drop down to a single person in the Business Office who 
fulfilled all these roles, but that was a demanding role. 

♦ Project Integration and Testing (I&T) Systems Engineer: LCROSS utilized NG’s staff to assure 
the spacecraft was being integrated and tested properly. Having project-level insight to these 
activities was essential in order to be responsive when issues surfaced, and to assure the 
requirements were being satisfied. The LCROSS PSE was plenty busy with other demands on his 
time, so having a dedicated project- level I&T SE would have given us great insight into decision- 
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making and would have substantially unburdened the PSE. LCROSS was able to get by without an 
I&T SE because of the following: 

• Having a very candid, honest relationship with the NG I&T SE, who would provide frank and 
accurate status of I&T activities. This NG I&T SE behaved functionally as the project I&T SE, 
helping to defray some of the burden on the PSE. 

• Having project liaisons stationed at NG to participate in the activities of spacecraft integration 
and testing was immensely helpful and productive. This role was not as policeman, but as 
facilitator between NASA and NG. The effectiveness of this role depended wholly on the 
individuals and their relationships. The LCROSS liaisons got integrally involved with the NG 
team, and because they were technically competent, they quickly attained the trust of the NG 
I&T team, even being given a dependent role in I&T. 

♦ Project Verification and Validation (V&V) Systems Engineer: LCROSS struggled with this 
activity as the project was heading into final I&T. Walking through project-level requirements and 
verifying them prior to launch involves an incredible amount of work which is very time 
consuming. The PSE owned this important activity on LCROSS, which impeded the amount of 
time he could spend on his other duties during the project phases that are most difficult for the 
Systems Engineer. Having a person who manages requirements and reports to the PSE would be 
very wise. . . if you can afford it. 

The small team staffing approach, based on our experience with LCROSS seemed to strike the right 
balance of resources and roles. Additionally, NG found great results by mixing experienced, veteran 
engineers with younger engineers. The experienced engineers had the pragmatism to know the right way 
forward and the younger engineers had enthusiasm and frankly didn’t know that we couldn’t possibly do 
it this fast! As noted above, I would staff some roles differently based on just how tough it got with this 
small team, to create a good core team. It is important to note that depending on the project size, the roles 
identified above do not necessarily have to be allotted to single individuals. Very small teams can 
organize in a similar way with team members wearing multiple hats. 

Team Collocation: Be sure to get your core team collocated wherever possible. Increasingly, virtual 
presence can substitute for collocation, but where feasible, actual physical collocation is the best. Having 
my deputy Project Manager, Business Manager, Contracting Officer, Project Systems Engineer (PSE), 
Project Scientist, Master Scheduler, and Project Control Manager all in the same hall with me, enabled 
easier and more responsive problem solving. For example, if I had a technical issue relating to the 
spacecraft, I could go ‘round-up my Contracting Officer (CO), the Contracting Officer Technical 
Representative (COTR), the PSE and his deputy and we could consult right there in the hall or in an 
office at a white board. Similarly, if there was a cost overrun issue by one of the Control Account 
Managers (CAMs), I could get the Business Manager, the Schedule Manager and the deputy PM and chat 
consequences and reserves right at that moment. 

Electronic means of communication are getting better all the time, but now everyone is inundated with 
electronic stimuli. Physical collocation affords a very real advantage over the other pulls on the team’s 
attention. Some have attributed the failures of notable missions in the past to communications 
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deficiencies associated with not being collocated, leading to fundamental misunderstandings. “If they’d 
been in the same room for any period of time, somebody would have realized you were speaking in 
different languages, and corrected that problem .” 1 



Figure 7-1 : South pole map with LCROSS targeted impact site in 

Cabeus crater. Image by LRO’s Diviner instrument. 

The Team Bubble: There is an interesting and somewhat unexpected problem that can occur with highly- 
successful staffing: the high-performance team. This sounds like a nice problem to have, right? Well, there 
is a very interesting dynamic which can occur with well-staffed, high-performing teams... . Forgetting 
where home is. This occurs in organizations where staff are matrixed to a project from a home 
organization (the line org). To further illustrate, it’s important to understand what staff matrixing is. The 
idea is that various line organizations are actually responsible for the employee and his development, 
which includes everything from timecards, training needs and the institutional requirements like 
perfonnance reviews. When a project appears, it does not usually become a line organization itself, but 
instead becomes a separate entity which is staffed with personnel matrixed to it from various line 
organizations. The PM comes from a line org where the PM’s reside; the PSE comes from the Systems 
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Engineering Branch; the Project Scientist comes from the Science Directorate, etc. These people from all 
these different orgs then fonn a team with a common purpose, and are likely collocated together in a 
building somewhere. No problem, right? True, but when this team starts really gelling as a group it starts 
to create a bubble around it which is defined by its common, highly-focused purpose. When this happens, 
the members of the team start to define their existence by their membership to the team and less to the 
home organization for which they actually belong. This bubble is a sign that the team has a highly-unified 
vision for where they are going, but may be losing touch with their home which can cause hostilities to 
form if there are ever conflicts between the needs of the project and the needs of the line organization. 

The answer to addressing this issue is clearly not to destroy the high-performance team! The PM can help 
considerably by staying in good contact with the various line organizations which are supplying project 
staff and making sure the line org’s needs are being addressed. This can include being good about 
providing the line orgs with the project’s assessment of perfonnance of their staff. The line org will find 
it is seeing less and less of their own people who are matrixed to the project and they will have less 
ability to comment and evaluate their perfonnance. The PM can help fulfill that line institution need. On 
LCROSS, my deputy PM, John Marmie was outstanding at being responsive to the line orgs, supplying 
feedback and recommendations for awards and promotions as appropriate. The PM can set the tone for 
helping his matrixed staff remember the importance of taking care of the needs “back home.” 

Team Staffing Adjustments: A high-performance team does not magically come about by finding great 
people, putting together in a collocated environment, and then they magically work perfectly together. 
Despite having a great team of people at the onset of LCROSS, as the project moved forward it became 
apparent that some of the team mates were not up to the task. This not only affects perceptions outside the 
project, but within it as well. Team mates can sense when someone is struggling to keep-up or is becoming 
a negative influence on the team. In one of the staff changes I had to make it was because the person 
couldn’t handle the incredible pace of a flight project; in another case, the person was technically very 
competent, but was not the right person to 
be leading the effort. In still another, the 
person was placing unachievable 
demands on himself, and the result was a 
physical manifestation of the stress 
leading to medical problems. Finally, one 
person was getting so frustrated by the 
challenging requirements that he was 
becoming mentally resigned to failure, 
and that was poisoning team morale. 

Each of these scenarios was very different, and yet each one required attention. As the PM, team health is 
your job. I had never removed anyone from their position before. This is not something you ever want to 
do. . .but you have to look out for the best interests of the team. This isn’t about you. These changes were 
incredibly difficult for me to do, personally, but having the willingness to make difficult changes, early 


As the PiVl, team health is your job. 

I had never removed anyone from their 
position before. This is not something 
you ever want to do... but you have to 
look out for the best interests of the team. 
This isn't about vou. 

M 
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on, with clear rationale and integrity, is a huge confidence-builder for both the team and the stakeholder 
community, providing that it is appropriate, free of agenda, and handled fairly. 

In three of the cases above I had to replace the persons and remove them from the project, and in the 
remaining case I moved the individual to another role on the project, better suited for his talents and 
skills. That was not to find a soft-landing, but was because I truly valued this person’s knowledge - he 
just couldn’t lead the highly visible activity. 

For the person I chose to simply move from a leadership role to a supporting role, I consulted with NASA- 
ARC management to see if this was going to cause any political problems, as this was a prominent role on 
any flight project. I was also hearing some voices of concern from different circles about the changes I was 
making; however, the NASA-ARC deputy center director told me that he wanted a person in charge of the 
project who can make the tough decisions and know when to make them, and he fully endorsed the move 
I was making. That was a moment of empowerment for me and helped me see what this PM job is really 
about: leadership of an effort for the effort’s sake. If I believe something needs to change on the project to 
make the project more successful, it is my job to assure that that change happens. 

So in summary, define a small team which fulfills the roles identified, plus any particular additions 
required by your circumstance, get them collocated together in one place as feasible, don’t forget where 
they came from, and be willing to make necessary changes in the best interests of the team. A small team 
size makes the decision process quicker and promotes skin-in-the-game individual ownership of 
responsibilities, while the centralized location promotes an efficient communications environment to 
enable the team to push forward. There is no one-size fits all answer to staffing and depending on where 
your project carries risk, you may need to trade resources in order to still fit in the box. In the end, you 
need to strike a balance between the stakeholder’s accepted risk tolerance and your resources to execute 
the mission. Finally, take the long view on the needs of the project. Be honest with yourself about how 
everyone on the team is doing and make needed adjustments with respect and purpose. 


End Notes 

1. Chaikin, Andrew, “Faster, Better, Cheaper: A Space Historian Takes Stock” Space. Com, April 19, 2000. 
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As a NASA Project, LCROSS fell under a programmatic authority and a technical authority (TA). The 
Agency organizes projects this way to assure that programmatic decision making and technical decision 
making can elevate concerns as required, for adjudication. The project manager represents the functional 
programmatic authority on the project while the project systems engineer (PSE) represents the technical 
authority. The PM still retains the overall responsibility for the successful execution of the project (both 
technical and programmatic), but there is a separate technical authority chain should circumstances 
require it. Agency authorities get updated from time to time, but during LCROSS, the PM’s reporting 
authority went up through the program office, and then up to the appropriate NASA Mission Directorate 
(ESMD). For the LCROSS PSE, the reporting authority went up through the NASA-ARC Chief 
Engineer and Center Director, then to the Agency Office of Chief Engineer (OCE). 

This separation is further assured by not allowing the PSE to charge time to the project-funded WBS, 
instead receiving labor funding through the Chief Engineer’s Office; the concept is that in organizing this 
way, the PM does not hold implied authority over the PSE’s assessment by way of influencing his salary. 
For LCROSS, this differentiation of reporting authorities was never a problem because the LCROSS PSE 
had a good programmatic sensibility, and the LCROSS PM was an engineer who understood the 
technical part of the mission. Coupled with great mutual respect for each other at a personal level, this 
meant that we ultimately came to consensus solutions on difficult issues without requiring elevation. 

“The fundamental objective of Systems Engineering is to successfully manage the technical content of 
the project to meet the cost and schedule constraints of the Program while ensuring that the technical 
content matches the expectations of the customers and ‘Stakeholders’” 1 . This definition of systems 
engineering illustrates why I believe the PSE has the most important job on a spaceflight project (yes, 
more important than the project manager). If there is a failure of systems engineering, the probability of a 
successful resultant mission is very low. That being said, given that LCROSS was designated a PM-led 
mission, the PM had ultimately responsibility for the overall performance of the project and mission, 
including both the programmatic and technical performance. This overlap affirms the importance of the 
PM and PSE working very closely together; they are both looking at the same prize, even if from slightly 
different vantage points, creating a natural tension in the roles. 

Given this interrelationship between the PM and PSE roles, it’s not appropriate for them to simply 
divide-and-conquer the management roles, where the PM addresses the cost and schedule and the PSE 
addresses the technical matters. These three facets of a project are so interrelated they cannot be 
decoupled into separate ownership. Both the PM and PSE need to be flexible and adaptable in evaluating 
technical decisions against a programmatic backdrop. 

There are a number of key areas in which it is essential for the PM and PSE need to have good alignment 
and agreement: 

♦ Team Makeup: It’s of course ideal to build a team with highly experienced individuals in order to 
reduce the sheer number of people required. We found on LCROSS that while having highly 
experienced team members generally costs more, having an experienced team working to solve 
problems more than pays for itself in team size minimization, and in fewer mistake-based go-backs. 
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Northrop Grumman is no small aerospace firm, is located in Southern California, and their labor rates 
reflect that, but the skill of the team they brought to the LCROSS table to design a spacecraft was 
impressive. The same was true, although in different ways, with the NASA-ARC, NASA-GSFC, and 
NASA-JPL team members, who were able to accomplish so much with so little. This expertise can lead 
to some strong opinions on matters, however, so it is important that the PM and PSE be on the same 
page guiding forward progress. 

The PM and PSE need to establish 

Technical Review: Technical reviews must be ^ of reviews . M ov je critics 

based on a common approach to risk manage- 
ment by the PM and PSE. Projects usually host <***e not allowed. Reviewers should 
an Independent Review Board (IRB) which not judge their effectiveness by the 

watches over the team’s development work on number of Criticisms thev Cast 
behalf of the stakeholders. The IRB tries to 

assure that the project is making appropriate progress through the milestones. The IRB itself is an 
excellent source of information and expertise (and are generally paid for by the stakeholders not the 
project), so make use of them. Since the IRB tends to be populated with both technical and 
programmatic experts, both the PM and PSE should establish good relationships and trust with the 
members of the IRB. The IRB members will be influential on matters of tailoring requirements with 
the stakeholders, including topics like the number and nature of fonnal reviews. The PM and PSE 
need to establish the tone of reviews. Movie critics are not allowed. Reviewers should not judge their 
effectiveness by the number of criticisms they cast. “[The] objective [in staffing the IRB] was to have 
people that would normally be selected to chair a post-flight Failure Review Board (FRB), where the 
objective is to find problems that could result in failure modes.” 2 In short, the panel’s objective and 
behavior should steer towards reducing the likely hood that this mission will end-up at a FRB. The 
PM and PSE should help to assure this is the thrust of the project reviews. 


♦ Setting the Tone: The PM and the PSE set the tone for the behavior of the team, and this further 
requires good alignment between the two. 

• Requirements Development and Control: A culture of strong requirements management is 
essential to keeping the work moving forward. This culture must be established beginning with 
the PSE who owns the requirements, but also with the PM, who must hold the CAMs 
accountable to living within their budgets, which are based on their performance requirements. 


• Quick decision-making: A short development 
cycle requires quick decision-making, which in 
turn means you cannot wait until you have all the 
information you need to make a decision (or put 
another way, you cannot wait until your confidence 
level is at 100%) - that will doom your schedule. 
Make informed decisions with what you know, and 
move on. If later facts lead you to refine or reverse 
your decision, then do so, but keep moving 
forward. 


You cannot wait until your 
confidence level is at 100% - 
that will doom your schedule. 
Make informed decisions w ith 
what you know, and move on. 
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• Few Absolutes: Since LCROSS 
was a Class D mission, the most 
risk-tolerant classification in the 
Agency, the idea of developing a 
traditional Incompressible Test 
List (ITL) did not fit the nature 
of our designated risk 
classification. An ITL is a list of 
a minimum set of tests that will 
be conducted on the spacecraft 
and systems, defined early in the 
project. The ITL concept is that 
you define a floor level of 
testing, below which you will 
not drop, regardless of what 
circumstances befall your 
project. In short, the ITL is 
supposed to protect you from 
excessive test compromises 
downstream, when the project 
may be in schedule or financial 
distress. This topic of an 
LCROSS ITL surfaced at the 
LCROSS Spacecraft 
Acceptance Review (SAR) with 
the reviewers asking the team to define the LCROSS ITL. I explained it didn’t make sense for 
LCROSS to have one since as a Class D project with a cost and schedule cap, we needed to have 
all options on the table, including modifying test plans as required to fit programmatic 
circumstances at the time. If you ’re not going to consider testing risk on a cost-capped Class D 
mission, when will you? The stakeholders came to agree that LCROSS could continue forward 
with its initial testing plans as its baseline, but there would be no ITL. If required, and per our 
risk management process, we could deviate from our baseline testing plan (including adding 
testing which we did right before launch), based on our programmatic risk context at the time. 

The PM and PSE are an important pairing on a project team because between the two of them, they 
answer for all the facets of a project’s performance. Given this importance, it is essential that the people 
who occupy the roles are not only compatible, but that they have good, mutual understanding of what 
each of them are responsible for and the needs of the other. The LCROSS PSE had an intuitive 
understanding of the demands placed on the PM and the reporting requirements therein, so he was 
already thinking in terms of supporting those responsibilities. The LCROSS PM understood the latitude 
that is needed by the PSE to survey an issue and work-out a path to resolution - within the programmatic 
framework of the project. Together, the PM and PSE are critical in paving the way on a project. 



Figure 8-1 : LCROSS Spacecraft after 

completing final testing and ready to ship. 
Image courtesy of Northrop-Grumman 
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End Notes 

1. Keith, C., “AFRL Streamlined Systems Engineering: Processes & Products for Success,” The Aerospace Corporation, 
February 5, 2007. 

2. “Company Capabilities,” Dubyn, S., Millennium Space Systems, 2010-09-09. 
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Lcross answered to a series of requirement which were owned by the stakeholders; in particular, Level 
2 requirements held by the program office and Level 1 requirements held by NASA-HQ. One of the key 
facets to successfully navigating a Design-to-Cost [DTC] project is to understand the key objectives which 
are essential to be met or the project is not deemed successful. This subset of requirements, which are the 
bare minimum requirements established to define a “successful” mission, were called, “ Minimum Success 
Criteria .” Highly-desired requirements, but with lower priority than the minimum threshold requirements 
are considered ‘‘‘‘Full Success Criteria .” Minimum Success Criteria requirements are of primary 
importance, while Full Success Criteria are of secondary importance. On LCROSS this was the 
framework by which all of our work was ultimately assessed. It is conceivable that some missions could 
have yet another set of tertiary requirements, which would be even less important than the full success 
criteria, but still highly desired. These could be called “ Extended Full Success Requirements, ” and provide 
a way to capture opportunistic goals, should the project and mission be able to accommodate. 

For LCROSS, the success criteria were essentially the following: 

♦ Minimum Success requirements: LCROSS requirements necessary to assure an impactor is sent 
into a permanently-shadowed crater. 

♦ Full Success requirements: Minimum Success requirements plus the LCROSS spacecraft is able 
to make in-situ water-ice measurements of the ejecta plume. 

♦ Extended Full Success requirements: Not defined for LCROSS, but these would effectively be 
Full Success requirements plus (for example) other interesting measurements made along the way. 
These would be opportunities to do other neat things if feasible, without compromising more 
important requirements. 

The reason I show the third type of success requirements is to illustrate how they might be put to use in a 
real project scenario. Whether in the design phase of the project or during actual mission operations on 
orbit, the project may find itself in a technical bind for one reason or another, and need to look for relief. 
As noted in a previous chapter, this relief could come in the form of consumption of schedule margin 
(schedule bucket), or cash reserves (cost bucket), or in consumption of some technical margins or systems 
capabilities (requirements bucket). The problem is that those options for getting relief are the same options 
that you’d go to if your full or minimum success requirements were at threat, and those are considerably 
more important to your project’s success. It likely makes more sense to trade-away some extended success 
capability rather than make use of project reserves or technical margins. If you find that you can dump all 
of the extended success requirements and it doesn’t fully relieve your current problem, then you need to 
look to each the full success requirements and the buckets and make a judgment what to do. This is very 
the right answer is highly circumstantial. You have to perform an assessment where the pain would be 
minimized. Maybe release of one, minor full success requirement would completely resolve the issue. 
Maybe it would be completely resolved with simply consuming 2- weeks from your schedule reserves. 
These are the trades and considerations that the PM has to make in conjunction with his team. 
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It is interesting to look at some hypothetical situations which can occur with particular success 
requirements. Given the LCROSS minimum and full success criteria, we had a couple of very-real 
situations where unexpected trades appeared. For example, given that our minimum success criteria only 
required that we send an impactor (of a given minimum mass) into the selected crater, and not take any 
measurements ourselves, we could completely abandon the LCROSS payloads! This means if during the 
design phase we found that we were badly busting our mass budget, and could not come up with another 
reasonable way to solve it, we could give-up on putting payloads on the spacecraft, thereby saving the 
mass... The same type of scenario could occur if we ran short on power budget. We could simply not 
power the instruments, saving that power to assure minimum success of getting into the selected crater. 
This was clearly very undesirable because we would be ceding all measurement of the lunar impact to 
others, such as possible earth measurements and other lunar orbiting assets such as LRO...but that is 
precisely what the success criteria are excellent at helping to define. 

One of the risks that LCROSS had to address was associated with the separation system which would 
allow the centaur rocket (impactor) to separate from the LCROSS shepherding spacecraft shortly before 
impact. This separation is what allows the LCROSS shepherding spacecraft to witness the impact of the 
centaur and capture all the important data. These separation systems are very reliable, but they are always 
activated within an hour after launch under normal use, thereby limiting their exposure to the space 
environment. For LCROSS, we would carry that separation system in orbit for over 100 days before 
expecting it to operate, which was well beyond its intended design use. Ground tests were conducted to 
evaluate the systems, but there were some anomalies with the test findings by the launch services program 
and we had to decide what to do. In the end we decided we’d just accept the residual risk because we were 
running out of time and the minimum success criteria was not at risk - only the full success criteria. If the 
separation system did not work, or partially worked (what we called the “dangling chad” scenario), we’d 
still pile-drive into the crater and succeed with minimum success. All that work we put into what would be 
otherwise perfectly functioning payloads, would be crushed in the crater with no data. Now this never 
ended-up being an issue because the separation system behaved as desired and the rest is history. 

In another scenario after LCROSS experienced an on-orbit 
propellant anomaly, fuel was at an absolute premium for assuring 
the minimum success criteria (striking the crater). There were some 
additional calibrations that the payloads team wanted to do with 
their instruments that would require orienting the spacecraft, 
thereby expending propellant. Before the anomaly, the answer was 
yes, and after the anomaly the answer was no - all because of the minimum success criteria. If we ran out 
of fuel needed to trim our journey into the crater, we’d still hit the moon, but not where we intended, 
leading to our being cast as a mission failure. We would have pursued full mission success at the expense 
of minimum mission success. This is another case where the instrument calibrations were fine and usable 
for collecting the needed data. 


Before the anomaly, 
the answer was yes, and 
after the anomaly the 
answer was no. 


71 


Chapter 9 — Managing the Success Criteria 


Defining the success criteria at the beginning of a project is effectively pre-negotiating with the 
stakeholders, what the off-ramping requirements priorities are. They define what the stakeholders would 
be willing to give up first and what would be last. While unpleasant to entertain, this is a very powerful, 
and useful tool for the project manager, and broader team, to understand at all times. 
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As discussed earlier, the LCROSS mission was not about pushing the limits of technology or testing the 
limits of performance. It was about doing as much as we could within the capabilities of our system. I 
once heard it said, “We want it all, but we know we need appetite suppressants. 1 ” Capability-driven 
missions like LCROSS are just as the name implies: you work to achieve the requirements by staying as 
much as possible within the capabilities of your system. This is very different from many science-driven 
NASA missions, where needs are defined and then custom-designs created to satisfy the science needs. 
Custom development is fraught with risk and can be costly by taking a lot of time in the design and 
testing phases. LCROSS was a design-to-cost 2 (DTC) project, working to cost and schedule constraints, 
the principal drivers for the project. By dealing as much as possible with existing designs, we had a set of 
capabilities with which to work, and that helped to contain cost and schedule (Figure 10-1). 
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Figure 10-1 : Design to Cost (DTC). 


The perfect incarnation of a capability-driven project requires few to no modifications to capabilities that 
currently exist. Everything is not only flight-proven, but is proven in the identical arrangement or 
configuration of its use on the project. In reality, real design-to-cost projects needs to carry sufficient 
reserves not only for the unknowns, but for the inevitable effort required to address the risk associated 
with expanding those capabilities where required. This concept ties back to the fill-level of the cost and 
schedule buckets. Of course, you should also have in mind descope options from the requirements bucket 
or even carry sufficient technical reserve to accommodate more mass or power needs as the design 
evolves. The key is setting up the appropriate bucket fill levels appropriate to the degree to which your 
project is going to deviate from the ideal DTC project. NASA provides guidelines for each phase of a 
project to provide aid in defining appropriate technical margins to carry, but the PM should rely on the 
technical team to understand the particular technical risks of the project and then balance that against the 
overall risk classification of the Project. 

Existing capabilities were the way for LCROSS. If something 
already existed - excellent. If something had already flown - 
even better. Given the LCROSS desire to be capabilities-driven, 

I want to illustrate some of the features and approaches taken on 
LCROSS to keep things as simple as possible. 

♦ Spacecraft structural bus. The LCROSS structure is based on the stout and simple ESPA ring 
described earlier (See Figure 10-2). The ESPA is a secondary payload adaptor designed to be 
inserted between the launch vehicle and its payload spacecraft, enabling various small payloads to 
be mounted on each of the ESPA ports, making use of excess launch capability. The ESPA was 
originally developed for the United States Air Force by CSA Engineering, measuring 60 inches in 


If something already 
existed - excellent. 

If something had already 
flown - even better. 
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diameter and 24 inches tall. It is constructed of aluminum, with a mass of 230 lbs and is able to 
support six secondary payloads each having a maximum mass of 400 lbs. The ESPA ring is very 
sturdy, able to support a primary payload of up to 15,000 lbs, which was perfect for LCROSS in its 
location in the LRO launch stack. LCROSS didn’t need to design a formal spacecraft bus because 
the ESPA had ample structural margins, and best of all, didn’t need to be qualified by the small 
Class D LCROSS project. Instead of using the ports to mount different spacecraft, LCROSS used 
them to mount different parts of a single spacecraft. One port supported the solar array, while 
another housed the batteries; another housed the avionics boxes and yet another supported the 
payload instruments. Best of all, the hollow center of the ESPA ring was the ideal location for the 
LCROSS propellant tank. A lot of design and test labor was saved by employing the ring as well as 
reducing structural design complexity, given it is such a simple piece of hardware; there is little to 
go wrong. . . which means that completing the design and testing is inherently lower risk. Look how 
that one decision bought down cost risk, technical risk, requirements risk, and schedule risk, 
beautifully executed by NG. 



Power Control 
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R5 Panel 


ESPA Ring 
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Tracker 


Command and Data 
Handling Electronics 
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Attitude Control and 
Communications Electronics 
R4 Panel 


Science Instruments 
R6 Panel 
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Fuel Tank 


Batteries 
R2 Panel 


Figure 10-2: NASA’s Lunar Crater Observation and Sensing Satellite (LCROSS) 

Spacecraft Bus Schematic. 
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♦ Propulsion System. LCROSS also leveraged existing hardware here. The propellant ta nk was 
actually a surplus Tracking and Data Relay Satellite (TDRS) tank, identical to the two tanks being 
flown on LRO. Making use of this proven ta nk meant the qualification testing was already 
completed for our intended use. 

♦ Power System. The LCROSS power system was based on the GSFC/LRO Power Control Electronics 
(PCE) design. Using hardware that was going to be qualified by another mission meant LCROSS 
could avoid duplicating the qualification effort that would have been required with a custom design. 
So with this verified design, LCROSS made its own copy and leveraged the investment the Agency 
had already made in the original development - further reducing technical risk. 

♦ Attitude Control System (ACS). The LCROSS ACS was also inspired by LRO hardware and 
software architecture, arranged in a similar single-strung design. The design includes a star tracker 
assembly, the miniature inertial measurement units, the coarse sun sensors, and propulsion and 
deployment electronics. LCROSS bought all its own hardware, but leveraged all the hours of 
development and even joined some of the LRO procurements in defraying some of the labor costs 
of the acquisitions. 

♦ Communication System. LCROSS communications were based on standard medium gain and 
omni antennae, as well as a duplicate of the LRO S-band transponder. 

♦ Payload systems. LCROSS employed Commercial Off The Shelf (COTS) instruments and some 
flight-proven instruments, such as our visible camera. We looked to well-established instruments 
from the commercial and industrial world to see if we could use them on the LCROSS mission. The 
instruments would ideally be ruggedized to improve 
their chances of survival in the LCROSS launch and 
space environments, confirming that fact by testing 
them in relevant environments (vacuum, temp- 
extremes, vibe, etc.). This was an interesting and 
synergetic approach, as the vendors were very 
interested in seeing their instruments get tested on the 
Government dollar, and were quite accommodating in 
providing support when we found issues. It was a 
classic win-win. In the end, most instruments did very 
well. There were some small issues which were easily 
handled such as one instrument test failed because a small bolt came loose during vibe testing. No 
adhesive had been applied to the bolt threads to help secure it in a dynamic loads environment. Once 
the adhesive was applied, the device passed testing just fine. In another case, an internal cable came 
loose in the instrument, because it was not staked down. We reduced the length of unsupported cable 
by staking it, thereby decreasing the cable strain experienced during the launch environment. 


The vendors were very 
interested in seeing their 
instruments get tested on the 
Government dollar, and were 
quite accommodating in 
providing support when 
we found issues. 
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The LCROSS payload included 
re-purposed instruments from 
motorsports, fiber-optic, carpet 
recycling, and beer-making industries. 


The selected suite of LCROSS instruments 
included a thermal camera (MID-IR1), 
which has been used in motorsports 
applications; Near-IR spectrometers (NSP1 
& NSP2) which are used in beer-making 
and carpet fiber analysis for assessing 
recyclability; UV visible spectrometers (UVS) from standard bench- top laboratory equipment; a 
visible camera routinely used in shuttle launch imagery; and Near-IR cameras (NIR-cam), used in 
fiber optic communications applications. This suite of instruments was a clever application of 
existing hardware repurposed for the LCROSS space mission. 

LCROSS was a “glue mission,” attempting to glue-together already-proven hardware, software, and 
Integration & Test (I&T) approaches. The project’s residual technical risk was the design effort of gluing 
the components together, as well as general workmanship issues, which are always present in a 

spacecraft build. The beauty of this gluing approach is 
that it plays directly to the schedule and cost risk 
framework of a mission like LCROSS. Risk is kept in 
check, as the simplicity makes it less likely that you will 
have to take time and pay people to remediate a problem. 
Here are some examples of how LCROSS glued mission 
elements together: 


LCROSS was a “glue mission,' 
attempting to glue-together 
already-proven hardware, 
software, and Integration 
& Test (l&T) approaches. 


♦ Payload Glue: LCROSS used Commercial Off The Shelf (COTS) instruments from various 
vendors. The first design task for the payload team was to get all these instruments communicating 
successfully with the spacecraft avionics. Our visible camera vendor had an interface unit that 
enabled many of their cameras to be connected to a single interface point. We thought of using this 
device to interface not only with the manufacturer’s own camera, but also with the other cameras 
and instruments from other vendors. If feasible, this would save both development effort and risk in 
developing custom, flight-ready black-box solutions for each instrument. Technically this unit 
could be the source of a single point failure, but the simplicity of the design fell with-in the realm of 
the cost/risk trade we were willing to make. 

As it turns out, we were able to work with the other instrument vendors in exercising this RS-422 
interface option, which enabled all of them to use this one data handling unit. The “glue” that was 
required was to make sure that the data format of the RS-422 stream was compatible between each of 
the instruments and the data-handler. Then the interface between the data handler and the spacecraft 
avionics needed to be glued. This solution proved to be very effective, since all interfaces were based 
on a common standard data protocol, saving untold hours in development and - later - in flight 
suitability testing. 

♦ Spacecraft Glue: Frequently the development of a spacecraft bus involves a custom design - one 
which addresses the particular needs of the mission; this is how a science-driven mission in NASA 
typically works. The problem, of course, is that this custom development is labor-intensive and 
requires verification for flight suitability (another costly item). LCROSS instead took an existing, 
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proven piece of hardware called an Evolvable Secondary Payload Adaptor, or “ESPA” ring, which 
was already designed for flight and suitable for our launch vehicle. It could carry multiple secondary 
payloads at six circular ports around the perimeter of the ring, while simultaneously supporting the 
loads from a primary payload mounted on top. The ESPA ring was specifically designed to make use 
of excess launch mass capability for multiple secondary payloads. LCROSS was the first mission to 
use the ESPA as the structure for a spacecraft with the various ports housing the various subsystems of 
the spacecraft. One port supported the solar array, another supported the battery panel, and another 
supported the payload instruments, etc. This design approach was very efficient because each of the 
port panels was identical in shape, and with an NG-designed universal secondary structure to attach 
them to the ESPA ports, we glued the spacecraft together easily. This not only saved time and labor 
during the design phase and I&T, but it also made for resilient assembly, as each of the panels could be 
assembled, wired and tested independently on stands next to each port until ready for final attachment. 
The rugged nature of the ESPA ring quickly enabled LCROSS as a secondary mission spacecraft. 



Figure 10-4: LCROSS Payload 

Observation Deck, “POD.” 
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Figure 10-5: LCROSS ESPA 
ring freshly machined. 


Figure 10-6: LCROSS ESPA ring with 

attached secondary structures. 


♦ 



Mission Operations Glue: We’ve all seen traditional spacecraft control rooms - the ones with 
large consoles, filling a very large room with wall-to-wall screens. When you take one look at such 
a room and the number of people staffing it, and it apparent that this is an expensive activity in both 
labor and infrastructure. Given LCROSS ’s cost-constrained position, we had to find an inexpensive 
way to implement a fully-functional operations room. LCROSS set up a humble control room with 
a series of PCs on a local secure network. We put together a robust, redundant, multiple-monitor 
system, running the same 
software that was used 
during Integration & Test to 
leverage training and 
investments made earlier in 
the project. This secure 
ground data system was then 
connected to secure 
communications lines which 
enabled the control room to 
send and receive data with 
the spacecraft. No backup 
commanding facility was 
afforded, another risk 
LCROSS deemed acceptable 
that fell directly within the 
Class D “little to no 
redundancy” realm. 


Figure 10-7: LCROSS Mission Operations Control 

Room (MOCR) at NASA-ARC. 
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End Notes 

1. Steve Draper, dPM Operations, US Army Brigade Combat Team, Aviation Week A&D Programs Conference, 2009-11-03, 
Phoenix, AZ. 

2. Atkins, K., “How to Plan and Manage Reserves Effectively,” IEEE Aerospace Conference Proceedings, 2004. 
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How much a project should keep in reserve is a function of the relative risk of the undertaking. There 
are many rules of thumb, but in the end those rules of thumb make certain presumptions about the 
relative difficulty and risk of things going wrong. A project which employs highly mature, reused 
designs and products is likely to carry a lower level of residual risk than completely new developments 
and the reserve position should be handled differently. The closer a design team gets to wholesale new 
development, the more development risk it encounters, and from the buckets point of view, the more cost 
and schedule reserves it takes to cover that risk. 


On LCROSS we didn’t codify hard threshold levels for reserve positions; we simply tried to stay with 
high heritage everywhere possible. That being said we did start the project with initial reserves of 15%, 
so we did make some relatively light presumptions about the risk we would encounter. While we didn’t 
have formal threshold levels for our reserves estimations, we did roughly follow the guidance below: 1 


♦ Existing design used in the same fashion: 5-15% reserve 

• This reserve covers the residual risk associated with your particular instantiation of the design. 
You may be using it exactly as it was used in another application, but yours is still a new 
application. 

• This is in family with the LCROSS overall, initial reserves position of 15% 

♦ Modified designs: 15-50% reserve 

• This reserve applies when you are entering some new territory with the design, so while the basis 
for the design may be proven, you are breaking that basis in some form, incurring uncertainty 
and new risk. 

• Some of the lower-maturity LCROSS instruments carried this level 

♦ New designs: 50-100% reserve 

• This reserve position is perhaps the most difficult to assess because there is little or no basis for 
the design. In this case the design is largely a new construct, and depending on the system 
complexity, you could be carrying moderate to significant cost and schedule risk. 


These numbers are very interesting as they relate to LCROSS: Recall that the LCROSS overall starting 
reserve position was ~15%. From the standpoint of design heritage, our project position lit nicely into the 
“existing design” category, which was what LCROSS tried to be wherever possible. Most importantly, it 
reveals that LCROSS tackling a new design would likely have been a pretty risky venture, given only 
15% in reserves. Independent of how much design risk you are carrying, it is important to recognize that 


reserves do not exist to support requirements creep or 
enhanced design features. Reserves exist strictly to 
address the uncertainty of the plan, including risks of 
which you are unaware - the unknown unknowns. The 
reserves exist to address realized risks, or issues, and 
can come from any of the three buckets, based on 
circumstances at the time. 


Reserves do not exist to support 
requirements creep or enhanced 
design features. Reserves exist 
strictly to address the 
uncertainty of the plan. 
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So how do you manage cost reserves? How do you know how well you’re doing and if you have 
opportunities with a given reserves position, based on where you are in the project phase? LCROSS 
employed “cost-to-go” (CTG) to monitor our reserves position over the course of the project. This was a 
very effective tool because it simply related remaining reserves to how much planned cost was left on the 
project. There are many different rules of thumb on what CTG threshold levels to use in evaluating a 
project’s condition. Regardless of the particular number, there should be a low bar under which you should 
consider your risk position getting dicey. In this range, you need to fortify your position so that you don’t 
run out of cost reserves. This could include trading other bucket items like technical margins or schedule 
reserves if feasible. In the worst case, descopes could be considered. It can be difficult to assess the relative 
value between different buckets when needing to trade between them as they work in different currencies. 
The schedule bucket has units of time, the cost bucket has units of dollars, and the requirements bucket has 
many different types of units: kg, Watts, %, DegC, etc. In some cases you can convert between currencies, 
for example with time. Time can usually be equated to dollars, given a known amount of work required, 
staffing levels and hourly rates, and needed procurements. However, technical trades can be more difficult 
as there may not be conversions between mass and dollars, for example. In these cases you have to use 
judgment on your technical risk-to-go. That is, you have to evaluate your existing risk list and see how you 
are doing with the requirements bucket and judge if there is one technical commodity in which you can 
afford to accept more risk. For example on LCROSS, we had ample power margins on the spacecraft as the 
solar array outperformed expectations and it was simply a robust design. This meant we could give-away 
some power if it helped us in some other way. Now it is not always possible to trade a technical metric or 
capability and turn it into cash. Sometimes you need to make a three-step trade wherein one technical 
parameter can be traded-away or reduced, which would enable the spacecraft build to occur more easily 
(quicker), which will in turn save money. . . The possibilities are endless. 

CTG management also requires a high bar number that needs to be defined. Above that level you are now 
in a very good cost reserve position, and should consider encumbering some of those reserve taking-on 
risk reduction activities - because you can afford to. 10% / 30% were the threshold bars LCROSS 
employed; these thresholds struck a good balance between financial concern and opportunity. The beauty 
of this approach is that CTG directly relates to your particular project position and forward tasks, as 
opposed to employing general rules of thumb. 

When LCROSS cost-to-go reserves exceeded 30% I would start looking to buy down risk somewhere, as 
this represented a cost opportunity. During this time we would consider performing additional spacecraft 
testing, but only as much as the schedule margin bucket would allow. As the CTG plot in Figure 11-1 
shows, we grew our reserves position in the first half of the project lifecycle and then started consuming it 
when we got above 30% availability. Then as we got near the end of the project we were more comfortable 
in allowing the CTG% to hover in the teens because we were retiring a fair amount of risk at that point by 
getting through spacecraft I&T. Put another way, our risk-to-go was small. Other than that period near the 
end of the project, LCROSS cost-to-go reserves never did drop below 10%. It is important to note that when 
you do get to the end of a project, since the remaining work is getting small and the reserves position is also 
getting small, the CTG calculation becomes less useful. The denominator in the calculation starts to become 
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Figure 11-1: Reserves tracking sheet. 


of the work, assigned values or liens to that work, and then carefully handled the remaining reserve through 
project close. I found that plotting a CTG% chart was a very powerful way both to understand where our 
project stood financially, and to inform stakeholders. It was a very intuitive chart, and good, bad or ugly, an 
excellent historical record of our project perfonnance. 

We were prepared to change our behavior if needed to keep our reserves position in check. If required, we 
would descope requirements, or rethink the relative effort to complete a task. Choosing the latter meant we 
were most likely buying up our risk somewhere, which may have been required to keep costs in line. One 
of the key ways to increase your reserves position through the execution of the project is to beat the 
baseline schedule milestones. Labor cost is a strong driver of mission cost, and every day that you can beat 
the delivery date is cash that goes back into the reserves fund. 

Design-to-cost, capabilities-driven thinking, requirements prioritization, and cost-to-go projecting were all 
approaches that LCROSS used to reduce its risk position. LCROSS was a Class D mission, but this doesn’t 
mean that it should simply accept risk if there are approaches which can improve the likelihood of success. 


End Notes 

1. Atkins, K., “How to Plan and Manage Reserves Effectively,” IEEE Aerospace Conference Proceedings, 2004. 
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Chapter 12 — Managing the Schedule 


Schedule management is important for any project, but particularly for an ambitious, schedule-driven 
mission like LCROSS. LCROSS’ budget was -20% of an equivalent, traditional mission’s cost and 
needed to be delivered in about half the time. To jump to the end of the story, LCROSS was able to 
develop a spacecraft & payload and prepare a mission team, from project ATP (Authority to Proceed) to 
spacecraft delivery at a SAR (Systems Acceptance Review), in just 29 months. In fact, the spacecraft was 
actually delivered in just 27 months (June 2, 2006 through August 28, 2008), if you consider NG didn’t 
get their NASA funding until two months after project start! 


Northrop-Grumman’s spacecraft project manager, Steve Carman, 
was was instrumental in that impressive spacecraft development 
pace, employing a highly effective schedule management 
approach. 1 In this approach NG acknowledged some of the 
realities of how people typically plan their own work, and some 
of the basic principles of good Critical Chain 2 management. The 
broader project used similar approaches in developing the ARC 
payload instrument suite and the ground data system for Mission 
Operations. Below are some approaches to schedule management 
that we employed on the LCROSS project. 


V 



Figure 12-1: Image courtesy of: 

ProductiveByDesign.net. 


Schedule Planning 


♦ Push team to delivery earlier 
than plan. “Work expands so as to 
fill the time available for its 
completion. ” This is kn own as 
Parkinson’s Law 3 . In reality, you 
want to drive the opposite response 
- to push moving pebbles from the 
“to-do” pile into the “done” pile as 
soon as possible in order to create 
some schedule reserve for 
activities which may run long. If 
this schedule slack is on the critical 
path, then the project completion 
date could be earlier, giving you an 
early delivery. An example can be 
found in the network diagram of 
Figure 12-2. Tasks ACF represent 
the critical path to completing 
deliverable 50, as there are 7 weeks 



Figure 12-2: Schedule Merge Point (Image courtesy 
of: ProductiveByDesign.net). 


needed to get from deliverable 10 to 50 (the longest path). Task F is scheduled to take 3 weeks to 
complete, but finished early in only 1 week (2 weeks early). Does deliverable 50 then complete 2 
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weeks early? No. Tasks BE require 6 weeks to complete, so deliverable 50 arrives 1 week early 
instead of 2 weeks early, with tasks BE no owning the critical path. It is important to understand 
this network diagram logic so you can take advantage of opportunities like this. In fact, if you kn ew 
in advance that task F was going to finish 2 weeks early, you might have sat down with the CAM of 
tasks B & E and see if they can trim 1 week off their delivery. If feasible, then tasks ACF and BE 
would each be 5 weeks, matching that of tasks AD and bringing in the deliverable 50 a full 2 weeks 
early. To make this effort effective, this process must continue on all the critical path feeders of a 
merge point, and then on all the merge points of the project critical path. In this way you continue to 
propagate the slack created along the project’s critical path, eventually pulling-in the overall project 
completion date. A given project IMS will have hundreds or thousands of merge points like these 
and those tasks carrying some part of the critical path are where the attention needs to be paid. That 
margin to the project completion or delivery date is then your hedge against future schedule risk 
(late deliveries, accidents, etc.), or for cost savings by delivering the system early (which ends the 
team’s labor charging to their respective WBS numbers). 


The natural human tendency of 
planners is to create a highly 
achievable schedule... one with nearly 
a 100% probability of succeeding - 
and mavbe some schedule slack 
on top of that just to make sure; 
that is human nature. 


♦ Shorten task durations. Shortening task 
durations obviously benefits the schedule, 
but how do you do it credibly? In most 
projects, the Integrated Master Schedule 
(IMS) is really just a collection of smaller 
tasks, as was shown in Figure 12-2, 
arranged in a sequential order by their 
respective owners or Control Account 
Managers (CAMs). When everything is 
put together, you have an end-to-end 
schedule and a total planned duration for that project. However, the natural human tendency of 
planners is to create a highly achievable schedule. . .one with nearly a 100% probability of 
succeeding - and maybe some schedule slack on top of that just to make sure; that is human nature. 
If every CAM did this, the overall IMS would be very conservative and quite a bit longer than is 
likely needed. Further, with this comfortable schedule in place, no one is motivated to beat that 
delivery. . .after all, why? Since the CAMs will naturally gravitate to a highly achievable (-100% 
confidence), try sitting down with them individually and ask them what the schedule would be if 
eveiything went perfectly. They may scoff at the idea because that will never happen, but what you 
are encouraging them to do is remove the risk element from their estimate; risk will be treated 
separately. In doing this, the PM is more likely to get an objective sense of what time is required to 
implement a task. Another approach might be to ask the CAM what his task duration would be with 
only 50% confidence (a 50/50 chance of not making the schedule). For example, the CAM’s 
original estimate for a task might be 4-months, but if he dropped down to 50% confidence, he’d say 
there is a 50/50 chance of getting it done in 2-months. As the PM, you would then need to decide 
how aggressive to get with this approach, as you have to keep the CAM’s skin in the game - he has 
to believe that what you capture in the IMS is something that can be achieved. Part of this buy-in is 
the second step: risks. What are those risks that were stripped-away in that first schedule, or in the 
schedule which only has a 50% probability of working? If the PM doesn’t account for these risks, 
then this process is simply a game of odds and you are hoping to get lucky. If, however, you assess 
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each of the risks that the knowledgeable CAM has about his activities, you can assign resources to 
them in order to decrease the probability they will become issues. Maybe there is delivery risk on a 
key component, or perhaps there is risk that not enough of the right skills are available to perform 
an environmental test. The approach taken is to create a credible baseline schedule and addressing 
the risks to that schedule in the team’s risk management process. Mitigations might include a 
change in procurement approach (shortening lead times), or staff augmentation (lessening work 
burden on the small staff). Another benefit of shortening tasks from the original 100%-confidence 
level is that it attacks the natural human tendency to get a slow start on activities. Now that you 
have shortened the tasks with some acknowledged risks, the owning CAMs are very aware of the 
schedule and are positively motivated to get going to avoid owning the critical path. 


I once told a task lead that his 
slack in the project schedule 
“is not your slack; it is the 
project's slack.. .that I manage.' 


♦ Capture and store project slack at the end: On a 

previous project I once told a task lead that his 
slack in the project schedule “is not your slack; it is 
the project’s slack. . .that I manage.” My point was 
that he would get the time needed to successfully 
complete the task, but any “extra time” on his 
delivery path was the PM’s to manage. I would take that extra time and try to move tasks to the left, 
thereby moving that slack out to the right, accumulating as overall project slack. This logic also 
applies to lower-level parts of the schedule, such as merge points where multiple activities feed a 
milestone task. One of those feeder tasks will be completed last, which means the other feeder tasks 
have slack. The PM has to try to work that local critical path to liquidate the slack on those other 
feeder tasks and move everything earlier, thereby gathering slack to the right of the merge point. 
Especially look for tasks which are dominating the critical path to a feeder point, as they will yield 
the greatest slack-generating return. 


Schedule Execution 

♦ Celebrate early completion. As stated earlier, a CAM usually has little motivation to finish early. 
In fact, some forces may encourage taking it easy. The PM ideally establishes a culture wherein 
CAMs earn a merit badge of sorts by pulling in schedule. The NG propulsion team members were 
rock stars during their system development because they were able to grow an enormous amount of 
schedule reserve out of their plans using approaches described above. There were some things 
which required consumption of some of the slack they had generated in their plan, but it was a 
small portion of the overall slack they had generated. 

♦ Manage your near-term merge points. The project’s execution is sometimes driven by matters 
beyond its control. It is important to keep on top of both risks and opportunities. Sometimes 
circumstances enable a task to finish early, but if you haven’t planned for the opportunity of an 
early-completing feeder task to a merge point, subsequent tasks may not be able to absorb an earlier 
start. As was the case in the schedule planning activity, during schedule execution keep on top of 
the dynamics and talk to CAMs who might need to know about early starts. This could include 
talking to line organizations to get resources sooner or it may mean attempting to accelerate a 
procurement. The latter might cost more, so you have to consider that cost versus the savings of an 
earlier start on other project activities. 
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♦ Slack consumption. In an ideal world, slack generated during the execution of a project would 
move the project delivery date earlier and people could be retired from the project earlier, resulting in 
liquidated cash savings from the plan. However, complex project activities carry both individual and 
systems level risk, not to mention the programmatic risks and schedule slack is most often used to 
remediate these issues when they surface. Early on the project, NG generated more than a month of 
schedule slack by using techniques described above. This slack was consumed in many cases by 
forces outside of our control, with the first being 
a very late delivery of the spacecraft processor 
from the manufacturer. As you can see in the 
schedule slack plot (yellow line in Figure 12-3), 
we went from nearly 40 working days of slack to 
4 working days in March 2007; all because of 
the late delivery of a single item. The slack then 
jumped back up the next month through our managing our merge points and replanning the need date 
for the processor. Part of the mitigation involved pressing on the supplier to improve the delivery 
date and part of the mitigation was parallel-tracking the flight hardware environmental testing 
downstream to grow more slack. We couldn’t have known these events would enfold as they did; 
however, once it surfaced, we consumed project schedule slack (the drop to 4 working days) and 
then found a new way to grow it back (the return to 36 working days). 

♦ Schedule Performance. As noted earlier, LCROSS was exempt from traditional Earned Value 
Management (EVM) reporting because of our approach to schedule and cost variance reporting as 
well as our submission of the monthly EVM reports from NG. The NG reports were included in the 
project-level reporting package backup material. The broader LCROSS project-level approach to 
monitoring project perfonnance was as noted, to monitor schedule noting variances on IMS tasks 
with the CAMs. The schedule manager would monthly track the CAMs work performance by 
getting their reports on progress, comparing that to the plan. As variances began to appear I would 
go and speak with the corresponding CAM and discuss status and corrective action. By measuring 
and managing to schedule variance, I still got to see derivative performance by the team before it 
became a problem (i.e., I could see trends coming prior to actual critical path problems). Each 
month I assessed the progress of primary tasks in each WBS against the IMS and reported the 
variance. It could be either positive (getting ahead of plan) or negative (falling behind plan) and I 
reported it on an easy-to-read single-page chart (see Figure 12-4). The chart was color-coded to 
facilitate quick inspection of team performance: 

• Blue: Comfortably meeting or beating schedule 

• Green: Performance matching the plan 

• Yellow: Performance is lagging - need to mitigate with CAM 

• Red: Significantly late performance - Schedule threatened 


We went from nearly 40 working 
days of slack to 4 working days 
in March 2007; all because of the 
late delivery of a single item. 
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Figure 12-3: LCROSS Schedule Reserve Tracking Sheet. 


When I saw one of the systems going yellow, I would detennine which lower-level WBS was driving it and 
talk to the appropriate CAM. In most cases the CAM was well aware of the lagging performance and 
already had plans in place to improve it. In some cases, however, the existing plan was simply too 
aggressive, and satisfying it would take Herculean perfonnance from the team, substantially growing 
technical risk. In these cases, we convened a discussion about how to achieve the broader requirements- 
driven goal. Occasionally this meant replanning interim activities so we could still achieve the baselined 
milestones. In other cases, however, we discovered some performance creep sneaking into the effort. This 
was not occurring from the top, but from within the team, as some members of the team were “improving” 
on a capability or were looking to automate a repetitive task, which was not required (a nice-to-have). 

♦ Watching for Performance Creep. A high-capability, engaged technical team seems to naturally 
want to make their system or product the best it can be. This sounds great and illustrates the dedication 
of the team, but this runs counter to project management. The PM needs to watch for performance 
creep, and it is tough, but this is a direct threat to the schedule, and therefore to cost performance. 
When we encountered this on LCROSS, I explained to the responsible CAM that we needed to 
carefully identify which efforts were tied directly to the satisfaction of derived project requirements 
and which ones went beyond the requirements. The latter needed to be binned separately and saved 
for potential consideration later, as the focus had to remain on requirements satisfaction only. 
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Figure 12-4: LCROSS Schedule Variance Tracking Sheet. 


The problem with enhancements or design creep is two-fold: It steals reserves and it can grow complexity. 
Once a task’s performance requirement is met and find there is extra time a team member could easily 
rationalize that he has the time to improve the product. In fact he may even feel it is important to do so. As 
noted previously, that is not his time to consume - it is a project resource which could be applied to 
another activity which is struggling. Further, adding extra features or capabilities to a deliverable can lead 

to it getting more complex because there are more options or 
more automated features. These features will later need to be 
verified and validated, requiring more V&V time beyond the plan 
and consuming schedule. This goes back to the concept that the 
PM owns the slack on the project. No activities should be 
undertaken that go beyond the requirements. “Good enough” is 
our mantra. Now I did discover cases where a CAM wanted to introduce a level of automation that was 
thought to ease the burden on the operators once we flew the mission. This kind of thing needs to be 
looked at very carefully, because the “improvement” is actually buying down operator risk on the mission. 
It goes beyond simply wanting to make it better and becomes a risk management topic to be considered in 
the cost/risk tradeoff. Is the time required to implement the augmentation, and its verification, sufficiently 


No activities should be 
undertaken that go 
beyond the requirements. 
Simplicity is our mantra. 
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buying down the downstream risk to the operations crew? Sometimes fixes are easy to decide, but others 
are not and should be brought before the Risk Management Board (RMB) for consideration. 

♦ Replanning versus Rebaselining. As a project moves through its execution, the schedule must be 
adaptive. This means circumstances and situations will change, requiring adjustments to the plan. 
Some conditions require minor adjustments while others call for wholesale changes, so at what point 
is the project schedule being “replanned” versus “rebaselined”? This is an important differentiation 
because rebaselining means the whole project construct has to be revisited versus simply adjusting 
schedule plans. Rebaselining could mean that the project budget and launch date need adjusting - 
not good things in the eyes of 

the stakeholders in a Performance variance from the project plan 

discussion with the program typically starts as a technical issue, which drives 

office, the Mission Manager a sc hedule delav, manifesting in a cost overrun. 

pointed to the one-line 

schedule as a means for differentiating when a “schedule replan” constitutes a “schedule rebaseline” 
in the eyes of the stakeholders. He didn’t want the normal adjustments to the schedule to become a 
burden to the project or the program office; doing so would simply create noise and unnecessary 
effort for the project. He explained that he would consider a rebaseline condition one in which the 
“taco chips” milestone triangles on the one-line schedule (Figure 12-5) significantly changed 
position. The beauty of this approach is that the one-line diagram is sufficiently coarse to give 
latitude to the project management team to address matters internally. 
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Figure 12-5: LCROSS one page schedule. 
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So we now understand the importance of managing the schedule, but it is just as important not to lose track 
of what really drives everything on most projects: cost. The schedule, however, is a principal driver of cost 
because labor is by far the most expensive component of a project’s budget. Some even say there is no 
such thing as materials costs because materials are principally driven by labor costs in their manufacture. 
Performance variance from the project plan typically starts as a technical issue, which drives a schedule 
delay, manifesting in a cost overrun . 4 When addressing a variance, be sure to work within your stated risk 
classification to properly manage to the expectations of the Project. These expectations can lead to 
uncomfortable trade-offs in which the stakeholders may not actually be comfortable with the 
consequences of their stated priorities, but that is why they are codified. The PM must continually assess 
the risk with each of these decisions. 


End Notes 

1. Carman, S., “Hyperion Lessons Learned,” TRW Program Managers Conference, December 5-7, 2000. 

2. Goldratt, E, “Critical Chain,” North River Press . 1997. 

3. C. Northcote Parkinson (Nov 19th, 1955). Parkinson '.v Law , Economist. 

4. Manzer, F., ‘Assessing Project Performance’ NASA-APPEL course, Strategy Bridge, Inc, September 13, 2011. 
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Chapter 13 — Managing the Complexity 


Although LCROSS was designated as a Class D mission, 
able to accept elevated levels of risk, the project team was 
not interested in investing their careers in a mission that 
had lihle chance of being successful. How do you keep 
risk low, despite being allowed to take greater risks, while 
fitting within the cost and schedule caps of the mission? 

LCROSS did this by holding the overall system 
complexity as low as possible. As the Air Force Research Lab (AFRL) has noted, “Drive out 
Complexity!! Keep It Simple Stupid!! Focus on simplest method to meet mission requirements. No 
points for elegance.” 

“Program lengths are getting longer due to increased complexity,” 1 largely coming from trying to 
squeeze more and more out of a given mission size. A system-of-systems is the result, where the 
spacecraft is not simply an assembly of parts, but an assembly of other systems. This inherently complex 
network of interlocking platforms and technologies requires the precise integration and unerring 
performance of disparate pieces. Put another way, as a system becomes more complex, it is more 
susceptible to human design and test errors, and less likely to achieve success. The key is to keep the 
mission small: “Engineering challenges for smaller spacecraft systems are generally less complex in 
structural design, launch vehicle integration, power/thermal balance, control algorithms, [as] spacecraft 
cost/complexity scales with mass & volume”. 2 

You can of course offset complexity risk by dividing and conquering the problem, breaking it down into 
smaller and smaller pieces that can be more easily managed. But with the increasing number of pieces, 
you also increase the systems engineering resources needed to assure that everything will come together, 

and the integration and testing 

For cost-capped, schedule-constrained missions, required to verify that all of it 
the right solution is to keep complexity in check, will work in the end. while this 

approach can counter the risk 

associated with a complex system design, it is no friend to your project budget or schedule. So for cost- 
capped, schedule-constrained missions, the right solution is to keep complexity in check - make the 
system as simple as you can. 

This idea of system complexity being related to the cost and schedule of missions (and principle in 
driving their overruns) is not new. A number of internal NASA studies, Government Accountability 
Office (GAO) reports and Aerospace Corporation papers center on this very topic. The Aerospace Corp 3 
has spent a great deal of effort on this subject, actually defining a “Complexity Index” (Cl) to give a 
project-level assessment of a system’s complexity, and therefore risk, to the project’s schedule and 
budget. The Cl is based on system performance such as, pointing accuracy, slew rates, number of 
articulations, number of thrusters, pointing stability, mass, power, solar array size and type, battery type 
and capacity, and a number of other factors. The index is normalized between 0-1 (0-100%). Dozens of 
past missions, both successful and unsuccessful, were studied to discern their complexity levels and were 
plotted against their allocated development time and budget. A fascinating trend becomes apparent, 
enabling other missions to assess where they sit against the body of past mission performance. 


Drive out Complexity !! 
Keep It Simple Stupid!! 
Focus on simplest method 
to meet mission requirements. 
No points for elegance. 
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Figure 13-1 shows a data trend of successful, impaired and failed missions, indicated by green, yellow, and 
red markers, respectively. There is clearly a trend line which can be drawn through the data which illustrates 
time and complexity ratios which can lead to successful missions. Below this trend line, mission’s start to 
have difficulties, either becoming impaired, or failing altogether. The key is to understand where your 
mission fits within this data. Activities can be successfully finished in very short amounts of time, but they 
better be very simple (low Cl) to implement - this type of mission lives in the lower-left comer of the plot. If 
however, the requirements get more sophisticated and the complexity of the task grows, for that same amount 
of development time, impairments and outright failures begin to surface as the risk was just too great. As an 
example, draw a horizontal line that starts on the Development Time axis at the three-year (36-month) level 
and carry it out to a Complexity Index level of 0.7 (70%). None of the missions in that part of the graph have 
been successful. The data suggests that this level of complexity is highly risky to accomplish in a mere three 
years. However, a year or more additional time for development would put you in the much better company 
of successful missions (moving vertically-upward on the graph). In this situation the project can either accept 
a high probability of failure, or you 
work with your stakeholders to get 
more time (and therefore money) to 
execute your mission to buy down 
your risk. On the other hand, you 
could reduce the complexity of 
your system by simplifying the 
requirements. Notice how the same 
three year development time can 
lead to success if the mission is 
simpler, (if the Cl were 0.6 instead 
of 0.7). Of course, it is not easy to 
make a change like this if you are 
already at your critical design level. 

This is why it’s important to make 
such an assessment early in the 
concept phase. 

A similar plot was made on project cost versus Cl. It again features a meaningful trend line that can help 
see where a project fits in the family of similarly costed and scheduled missions. These assessments were 
applied to a number of the NASA Faster, Better, Cheaper (FBC) missions of the late- 1990s and their 
failures were in line with these complexity-based families of curves. As noted in previous chapters, the 
success of the early FBC missions led to more sophisticated subsequent missions with increased 
complexity. Without corresponding programmatic relief on these subsequent missions (cost and 
schedule), risk of failure grows. 
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Figure 13-1: LCROSS Schedule Complexity Diagram. 
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So did LCROSS use this approach? Indirectly. Early in mission development, the ARC Chief Engineer 
performed a complexity assessment of the LCROSS mission and plotted it on top of these families of 
curves. LCROSS was assessed at a ~0.6 complexity level, which was aided by the fact that there are no 
deployables or articulations on the LCROSS spacecraft, no reaction wheels, ample power margins, etc. 
The number of thrusters drove the LCROSS Cl up a bit, as did the fact that LCROSS employed a Li-ion 
battery - relatively new technology for spaceflight missions of the time. If you study the curves for our 
$79M cost and ~30-month development time, you can see we were on the fringe between the zone of 
successful and unsuccessful missions. This concerned us a little, but considering our Class D risk 
designation, it was actually exactly where we should have been. Of course, assessments like these are 
imprecise, as we are talking about families of curves. Lurther, this assessment is based on the performance 
of other missions which may have executed in different enviromnents, cultural and otherwise. However, 
this is a powerful tool to help understand where your project’s risk position sits relative to the on- average 
mission performance of your predecessors. This information could provide context that could be useful 
with both the stakeholders and the team. 


End Notes 


1. Ron Hornish, VP/GM Rockwell Collins, Aviation Week A&D Programs Conference. 

2. “SMALLER is Better: Technical Considerations for Operationally Responsive Space,” Lt. Col George Nagy, USAF, 

A1AA 24th Small Satellite Conference; Logan UT, 8/10/2010. 

3. “Perspectives on NASA Mission Cost and Schedule Performance Trends,” David Bearden, Aerospace Corporation, 

6/3/08, GSFC Symposium. 
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Recall that given the tough set of constraints in mass, schedule, and budget for LCROSS, the Agency 
classified LCROSS a “Class D” mission. Also recall that the ESMD Associate Administrator made it 
clear he was willing to accept additional risk for LCROSS, and in fact felt that this type of mission 
should be an important part of the Agency’s portfolio. So what is this Class D designation, really? 

NASA sorts (bins) all its spaceflight missions into four categories based on the Agency’s risk tolerance: 
Classes A, B, C, and D. This classification system has origins in the old Department of Defense (DoD) 
Military Standard DoD-HDBK-343. NASA has tailored these DoD classifications in a “Risk 
Classification for NASA Payloads” NASA Procedural Requirement (NPR 8705.4) 1 . 

NASA Class A missions are risk intolerant, characterized as “high priority, very low risk.” These tend to 
be either large, expensive missions, or manned spaceflight missions where human lives may be in harm’s 
way. Class A missions require generous technical margins when first formulated, and are well served by 
having good schedule slack and generous reserve dollars in order to provide the highly-redundant 
systems and extensive testing needed to assure that requirements are satisfied reliably. In short, failure is 
not an option and all three “buckets” need to be pretty full, which leads to elevated cost. 

Class D missions are at the other end of the risk spectrum. A Class D mission is the most risk-tolerant 
mission in NASA, characterized as “low priority, high risk.” Class D safety is treated no differently than 
a Class A mission, but mission assurance is. In fact, NPR 8705.4 states, “Medium or significant risk of 
not achieving mission success is permitted.” This type of mission is permitted to fail. This risk 
classification is typically applied to small missions that are constrained in some way, making it harder to 
assure mission success, and therefore the Agency allows these missions to accept more risk. It is 
anticipated that some or all of the mission requirements may not be met. 

Since LCROSS was cast as a Class D mission, we lived within the somewhat vague framework described 
above. The project was cost-capped so we had to watch the cost bucket very carefully since we started 
with only 15% cost reserves. LCROSS was also schedule-constrained in order to assure making the 
launch opportunity with LRO, so the schedule bucket was similarly handled with care. However, the 
Class D construct did allow LCROSS to waive requirements or take additional risk as necessary to fit 
into the schedule and cost constraints. That meant we had room to work with the requirements bucket, 
should that have been necessary to address a problem during development. 

This approach is also how the Level 2 program office viewed working with its L2 requirements that were 
levied on the LCROSS project. They established “minimum” and “full” success criteria to establish 
priorities when requirements trades had to be made. The program office L2 requirements document 
specifically listed, by number, the requirements that were minimum success criteria, leaving the rest as 
full success criteria. This effectively told the project, “If you are forced to start dumping some of your 
requirements, here’s how we’d like you to prioritize them.” Having requirements off-ramps defined early 
in the mission execution is important. Up front concurrence on acceptable ways to make contingency 
trades, saves time and heartache later. This approach well suits the Class D context, but the open question 
is, does the Agency really have the belly for this behavior and approach ? 
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Here is what NPR 8705.4 says about Mission Classification: 2 

Table 14-1. NASA NPR 8705.4 Mission Classification Definition. 


Characterization 

Class A 

Class B 

Class C 

Class D 

Priority (Criticality to Agency 
strategic plan) and acceptable 
risk level 

High priority, very 
low (minimized) risk 

High priority, low 
risk 

Medium priority, 
medium risk 

Low priority, high risk 

National significance 

Very high 

High 

Medium 

Low to medium 

Complexity 

Very high to high 

High to medium 

Medium to low 

Medium to low 

Mission lifetime (primary 
baseline mission 

Long, >5years 

Medium, 2-5 years 

Short, <2 years 

Short, <2 years 

Cost 

High 

High to medium 

Medium to low 

Low 

Launch constraints 

Critical 

Medium 

Few 

Few to none 

In-flight maintenance 

N/A 

Either not feasible 
or difficult 

May be feasible 

May be feasible and 
planned 

Alternative research 
opportunities or re-flight 
opportunities 

No alternative or re- 
flight opportunities 

Few or no 
alternative or re- 
flight opportunities 

Some or few 
alternative or re- 
flight opportunities 

Significant alternative 
or re-flight 
opportunities 

Achievement of mission 
success criteria 

All practical 
measures are taken 
to achieve minimum 
risk to mission 
success. The 
highest assurance 
standards are used. 

Stringent assurance 
standards with only 
minor compromises 
in application to 
maintain a low risk 
to mission success. 

Medium risk of not 
achieving mission 
success may be 
acceptable. 
Reduced assurance 
standards are 
permitted. 

Medium or significant 
risk of not achieving 
mission success is 
permitted. Minimal 
assurance standards 
are permitted. 

Examples 

HST, Cassini, JIMO, 
JWST 

MER, MRO, 
Discovery payloads, 
ISS facility class 
payloads, attached 
ISS payloads 

ESSP, Explorer 
payloads, MIDEX, 
ISS complex 
subrack payloads 

SPARTAN, GAS Can, 
technology 

demonstrators, simple 
ISS, express middeck 
and subrack payloads, 
SMEX 


LCROSS quickly discovered something interesting about the Class D construct. While the definition is 
adequately expressed in the cited NPR, there is no reference to this risk context in other NASA policy 
documents. The Class D methods we employed on the project ended up violating other NPR 
requirements due to self-inconsistency within the NPRs. Throughout project execution, LCROSS would 
bump into walls associated with these discrepancies. This ambiguity, or in some cases contradiction, 
became the Project’s to resolve with the stakeholders every time we found one. Navigating these 
instances required a fair amount of labor to work through the discussions with the stakeholders, having 
the unexpected result of burdening the Class D mission. 
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Another unforeseen angle on Class D was the extensibility of mission class designation onto the 
spacecraft contractor. Northrop Grumman knew that NASA had designated LCROSS as Class D, but this 
may or may not have been in alignment with how the spacecraft contractor operated with its own 
stakeholders. Once LCROSS was given the green light, the NG part of the team rediscovered their own 
equivalent of a Class D approach within their existing, approved corporate processes. The approach 
streamlined the oversight and approval processes, which helped them to come into NASA Class D 
alignment; this did, however put them in an interesting position: is a corporate entity willing and able to 
take the same risks of failure as the Agency? Of course, none of us signed up for this LCROSS venture to 
see it fail, but the nuances of precisely how to operate and how to define what is sufficient, while staying 
in the cost and schedule boxes, was challenging. For LCROSS, the Government directly assumed the risk 
by awarding a Cost-Plus contract with NG. Some stakeholders wondered why a fixed-price contract 
wasn’t awarded given some of the high-heritage of the LCROSS design, but such a contract means places 
all the risk on the contractor, thereby forcing their corporate position to be inherently more conservative, 
including placing margins on all the cost numbers for contingencies. We would have never made our cost 
constraint. With a cost-plus contract NG was able to be unencumbered with the what-if risk and simply 
do what made sense. For a project to live within fiscal means, all parties need to work within that 
framework; without the corporate buy-in on the LCROSS approach from NG management and the 
challenges posed to each CAM, LCROSS most likely would have not financially succeeded, regardless 
of the amount of contractual oversight. 

Given such ambiguities, one of the burdens of the LCROSS project was being a modem pathfinder for Class 
D for many at NASA-HQ. The advocacy required to win over the stakeholders to the project’s proposed way 
of doing things was very time-consuming. In the case of the 

NASA stakeholders, we pitched an approach that involved more “Wf hail to Crtiltt a waiver 
tailoring than waiving. We found that waivers evoke questions (q (ht Waive I' process.” 
which go something like this, “This procedural requirement is in 

place because past experience shows that this is a wise thing to do. . .so, justify why you do not feel it is wise 
for your project.” Just try to make that argument.... The NPRs reflect the collective experience of the 
Agency, across many missions, over many years. The problem is that the original purposes or rationales for 
levying certain requirements have been severed from the requirements themselves, and without this 
connection, a team might follow a set of rules that doesn’t apply to their situation. 

The Agency NPRs are meant to provide consistency across projects, but are not universally applicable to 
all types and classes of missions. Waiving those requirements is a tall order because it encourages a 
philosophical discussion about detailed implementation approaches, with multiple parties, who 
themselves may not be in agreement definitions of risk tolerance. A small, fast team does not have time 
for this. We learned on LCROSS that tailoring an existing requirement was a far more pragmatic way to 
tackle the problem. We tailored our construction of documents and artifacts and even our review process, 
which will be discussed later. 
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Northrop Grumman fortunately had an existing 
methodology which aligned well with the NASA Class D 
definition, but required a similar amount of effort in 
convincing their corporate stakeholders that their 
approaches were appropriate and acceptable. Their mission 

assurance organizations in particular required a lot of coaxing. NG employed tailoring and waiving as well. In 
fact, my NG spacecraft manager, Steve Carman used to say, “We had to create a waiver to the waiver process.” 


We had to expend monies 
and schedule satisfying 
another mission's risk position. 


Another feature of this Class D philosophy 
is what I call “class warfare.” LCROSS was 
a Class D mission, but the Agency hadn’t 
considered what this meant in the context 
of the sister mission with which it was 
flying (LRO), or the launch vehicle (Atlas). 

How do you manage integrated mission 
risk when LCROSS is designated Class D, 

LRO is designated Class B/C, and the Atlas 
launch vehicle is arguably designated Class 
A? We found out what might be suspected: 
the lowest common denominator rules the 
day. In the case of structural margins, for 
example, LCROSS had done some analysis 
calculations that showed generous margins 
on natural frequencies of the propulsion 
tank and the secondary structure. Atlas and 
LRO concurred that the analysis provided 
good numbers, but wanted additional 
verification based on more than just 
analysis. Because this level of verification 
was necessary for LRO’s designated risk 
position, LCROSS was forced to conduct 
testing on the structure and propellant tank 
to verify the analysis for others, as the 
LCROSS Class D designation did not 
require such a test. In essence, we had to 
expend monies and schedule satisfying 
another mission’s risk position. Since this 
was truly a “beyond scope” activity for 
LCROSS, we successfully argued that the program office should pay for it. “Class creep” also was something 
to watch-for as the closer we got to launch the more risk-averse the stakeholder community seemed to 
become. It can be difficult to prevent comforting testing from being introduced as you near the end. 



Figure 14-1 : LRO stacked atop LCROSS with the 

two PM's, Daniel Andrews and Craig Tooley. 
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As you will soon see, while we could accept additional levels of risk, we preferred to lower our net 
developmental risk in the first place. If you design your system to be low in complexity, you are decreasing 
the likelihood of needing to pulling from margins in your buckets. If a procurement is taking longer than 
planned, thereby threatening schedule bucket reserves on your schedule-driven mission, you can consider 
preserving some of that schedule slack by reducing some of the unit level testing you were originally 
planning on, because your system complexity is low. It may well be sufficient to rely more on system, 
subsystem or box-level testing since your built-up system has remained relatively simple. Doing so may 
come at the expense of more technical risk since an undiscovered problem may not show up until later 
testing; however, if your system is low-complexity in the first place, that risk might be worth the trade. 

Does this feel uncomfortable? It is understandable because so much risk could be retired at this lower 
level of testing, it just doesn’t feel right wait and allow potential gotchas to bite you downstream in the 
schedule. But what if that unit-level device or box has been proven on a previous mission, used in the 
same way you plan on using it? What if the card on which you reduce testing is easily removed or 
replaced from the avionics box, making the later discovery of a problem not so bad? The important point 

is to evaluate these options; the particular answers will be 
different for each mission and scenario. This approach 
requires judgment and a good understanding of relative 
technical risk. The key is to find ways to keep that risk in 
check, even if your designated risk classification allows 
you to take those risks. 

With all this discussion about accepting elevated levels of risk, however, it is important to realize there 
are two sides to the risk coin. Opportunity is the flipside of risk. The LCROSS team repeatedly impressed 
me with their ability to work around a “brick wall” and move forward. Many times these problems 
resulted in opportunities that ended up benefiting the project. LCROSS has been “opportunistic” when 
confronting risk or issues. Here are some examples where a problem or issue turned into an opportunity 
for the team: 

♦ OPPORTUNITY: Bus Bandwidth - As noted in a previous chapter, shortly after the mission was 
selected, we began digging into the avionics hardware design and discovered a limitation we hadn’t 
previously understood. We had planned to use a single data bus for transferring both the spacecraft 
operational data and the instrument science data. This accorded very naturally with our capabilities- 
driven approach. However, we found out that the 1553 data bus could not handle all that data traffic, 
so we looked for another way to handle the instrument data. Design a new bus? Hardly; that would 
be fraught with cost, schedule, and technical risk. . .remember our capabilities-driven approach? We 
ended-up finding a RS-422 capability built into the avionics design that we had not planned on 
using, and we began to study the possibility of making it a dedicated data bus for the science 
instruments. This represented a change from the original plan, and would therefore cost some 
additional time and money to implement in software, but it still played strongly to heritage since this 
bus already existed - this was our first major judgment call during the project. Now, in order to make 
this idea work, the instruments would need to be able to work with the proposed RS-422 bus. The 


The LCROSS team repeatedly 
impressed me with their ability 
to work around a “brick wall" 
and move forward. 
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payload team would have to create some sort of “black-box” to process each of the payload 
instrument signals to make them RS-422-compatible. As it turned out, a black box would have been 
necessary to get all the instruments to be compatible with the originally-planned 1553 bus anyhow, 
so there was work to be done in either case. 

Some of the instruments were already RS-422-compatible with no additional work (the ideal 
scenario), but some were not, or employed different flavors of the serial protocol. Here a second 
opportunity appeared. Our visible cameras employed the RS-422 standard and the manufacturer 
also had a data handling unit (DHU) product which would take those camera RS-422 signals and 
process them for usable telemetry on the data bus. If we could get the other instruments (from other 
manufacturers) to be compatible with that DHU, we could make the DHU the “black box” interface 
between the instruments and the data bus. We ended up taking this approach with fine success. It 
enabled one-stop shopping for interfacing all the instruments, which was cleaner and required 
fewer vendor mods than building from scratch since the DHU was already flight-proven hardware. 
All of this resulted in a lower-complexity solution, and remember that means lower cost and 
schedule risk. 


♦ OPPORTUNITY: Co-Manifest Design Constraints - A co-manifested mission (one which is 
launched with another mission on the same launch vehicle) involves an inherent schedule race, in that 
both of the spacecraft need to cross the finish line at the same time. However, during development one 
will inevitably lead the other in one area or another. Given the natural structural interaction between co- 
manifest missions, this could become problematic. An example: with LCROSS, we were attempting to 
locate our payload instruments within the center of our spacecraft (ESPA ring) above the propulsion 
tank. The problem was 
that LRO’s main engine 
thruster bells projected 
into the payload adaptor 
between the two 
spacecraft, and their exact 
locations would not be 
locked down until long 
after LCROSS needed to 
commit to a design. Since 
LCROSS could not wait 
for the LRO details to 
solidify and still make its 
own schedule, it would 
have to wait for LRO’s 
final design location, 
right? No. Here’s where 
the opportunity came in. 

Given this dilemma, we 

decided to look to an Figure 14-2: Space Launch Complex 41 (SLC41) at Cape Canaveral 

entirely different in F|orjda Launch sjte of LRO /LCROSS. 3 
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solution. Was there a place we could put the 
instruments that would avoid this interference 
risk altogether? It was from this thinking that we 
found a great solution for locating the instrument 
suite outside of the ESPA ring on a dedicated 
panel. In fact, this constraint led to moving all 
the internal electronics of the whole spacecraft 
from the interior area of the ESPA ring, to 
placing directly on the radiator panels. This 


We reduced the amount of wiring 
required, reduced the overall mass 
of the payload suite, and reduced 
the labor needed to accomplish the 
build, all while simultaneously 
improving the serviceability of 
the entire payload suite. 


eliminated complex heat pipe assemblies and 
simplified/shortened the wiring harness which would have had to be longer to make interior 
electronics serviceable. This design opportunity which was bred from a constraint reduced the overall 
mass of the spacecraft, reduced the labor needed to accomplish the build, and improved the overall 
serviceability of the spacecraft. In short, the overall complexity of the system was greatly reduced, 
thereby reducing risk. 


♦ OPPORTUNITY: Schedule Changes - The spacecraft development was out-performing the plan, 
delivering very early. This sounds great and of course it is, but other activities are tied to the 
delivery of the spacecraft (remember the problem of feeder paths to merge points?). This early 
delivery broke schedule synchronicity with Mission Operations Systems (MOS) development at 
NASA-ARC, crushing our ability to support a spacecraft-MOS end-to-end (E2E) test after 
spacecraft Integration and Test (I&T). We could choose to keep the spacecraft team around waiting 
for the originally-planned date, but that was not cost-effective, so the E2E testing at NG was 
cancelled. However, this turned into an opportunity: Conduct E2E testing at the Cape, where not 
only this would occur later in the flow (when MOS would be ready), but actual operational 
scenarios could be tested via true through-the-air test of the antennas via DSN station, back to the 
Mission Operations Control Room (MOCR), a much higher fidelity test. Further, this test would 
capture residual shipping risk since anything that might have gone wrong during shipping would be 
revealed in the E2E test. Thus the overall technical risk was driven way down. 


Now for full disclosure: if we hadn’t had time at the Cape to perform this test, it would never have 
been completed - recall we were intending to ship-and shoot, wherein we finish the spacecraft 
build and test, and ship it to the site ready to shoot. As it turned out, the launch manifest got 
changed and our launch date got moved while we were at the Cape and provided an opportunity to 
conduct the test. It had the additional benefit of keeping the team fresh and ready as we headed 
toward launch. But every test conducted carries risk, and adding a test adds risk due to human error, 
either in understanding how to connect the ground support equipment, or in procedural errors with 
the spacecraft. Nothing is ever simple. 

♦ OPPORTUNITY: Running too fast - During spacecraft I&T, the team had been running fast and 
was getting tired; mistakes started showing-up, increasing risk. We had good schedule margin, but 
cost was a driver. A three-day holiday weekend was coming, which meant people would be paid 
premium to continue working through it. Holding schedule saves money since you can let people 
off the project on time (or even earlier), but since holiday and weekend pay was a premium, and we 
had schedule slack to give, it made no cost difference to simply take the long weekend off. So by 
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intentionally letting the team go have their long weekend, they could decompress, “burn some hot 
dogs” over the weekend, and come back fresh at no additional cost. 

The key on a capabilities-driven, cost-capped mission like LCROSS is to keep it simple, and monitor both 
sides of the risk equation (risk and opportunity). This also means watching for opportunities that may 
carry hidden risk that offset their value. LCROSS had such an opportunity near the end of the mission. It is 
very important to understand that this opportunity presented itself after a propellant anomaly on orbit - 
which completely changed the cultural landscape for the mission and the stakeholders. This is discussed in 
great detail in a subsequent chapter, but the theme of risk management is the same. 

This late-arriving opportunity came in the form of testing a new software script on LCROSS, while it 
was on orbit, to demonstrate that it was viable and safe. Specifically, we had an opportunity to run a 
script which would test pulling data from various RAM memory locations on the spacecraft computer. 
Validating such a tool would be great for helping with diagnostics while on orbit, in order to see what 
was happening with the spacecraft, because you could probe any memory location from the ground. This 
memory probing tool had been run before on the ground with the simulator and with the spacecraft prior 
to launch, but never while in flight. The benefit of testing this script while on orbit was that the Technical 
Readiness Level (TRL), or maturity, of this software would be elevated and more easily made available 
for future mission use, perhaps even LCROSS if needed over the remainder of the mission. The 
downside, of course, was the possibility of unintended interaction with the flight software, resulting in 
unintended impacts to spacecraft attitude and functionality. So what to do? 

In the end, I decided against doing this test because it didn’t pass my prudence test, another way of 
stating that the risk trade did not add up for me. This was indeed the perfect opportunity to test the 
software and mature its TRL, but the mission team had just come off of a major anomaly in which we 
could have lost the spacecraft. All eyes were on us. We had implemented some changes on the spacecraft 
as part of the recovery from the incident, and made some operational adjustments. We were also very 
near the end of the mission, when there would be less time to recover from an unintended situation 
brought on by the execution of this script. Despite our Class D designation, we had come too far and 
recovered from enough off-nominal conditions that the right thing to do was allow the remainder of the 
mission to be uneventful, all the way into the moon. 

While we lost an opportunity to advance the maturity of what might have been a very clever script that 
could help future missions, the relative uncertainty with its execution made it a “no-brainer” to leave it 
off the spacecraft. It would be tough to explain why the LCROSS mission had failed over the testing of a 
potentially useful, but unnecessary script. . .there would be no adequate defense. 
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End Notes 

1 . http://www.hq.nasa.gov/office/codeq/doctree/87054.htm. 

2. http://nodis3.gsfc. nasa.gov/displayDir.cfm?t=NPR&c=8705&s=4. 

3. Image Approved for Public Release; Dirk Schreier, LMCO, 1-29-08, Does Not Contain Proprietary Data. 
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A mission anointed with a Class D designation needs to be prepared to accept elevated levels of risk... 
and potential failure. Are the stakeholders properly prepared for this reality? Is the public? On LCROSS 
we found a mixed bag. Some stakeholders saw LCROSS as a grand experiment in this Class D world - 
wanting to intentionally do only what is required under Class D in order to see if the paradigm works. 
Others felt very differently and considered LCROSS an experiment in risk management. The public, on 
the other hand never really had the right feel for the class D nature of this mission; we needed to do a 
better job educating the public. These differing views of LCROSS’ programmatic environment only 
made the job of LCROSS project management more complex; it was clearly a pathfinder. The pathfinder 
role meant that part of our responsibility was helping the stakeholders manage their own portfolio. 
LCROSS worked closely with both the program office and the mission directorate, as LCROSS and LRO 
helped to define their organizational concept of operations, being relatively new organizations 
themselves. The LCROSS pathfinder position gave us the opportunity to strongly influence the modem 
definition of a Class D mission within the Agency. 

A Class D mission like LCROSS is difficult because 
the trade space is vague, but this also means there is 
opportunity. LCROSS had the flexibility to manage 
risk , not eliminate it. To be able to operate in that mode 
though, the stakeholders needed to get comfortable 
with this idea of risk acceptance. Many of us on the project felt that while Class D missions can accept 
the highest levels of risk, they actually require the most rigorous risk management. Over the course of the 
project’s many Risk Management Boards, we received accolades for the frankness of our approaches and 
the evenhandedness with our evaluations. If everyone is working to the same risk framework, it becomes 
easier for everyone to get on board. 

But for NASA to be able to support missions with a greater risk of failure, it needs to actively manage its 
stakeholders: the White House, Congress, and the public. Simply put, Class D missions have a higher 
probability of failure... period. The message to the stakeholders, however, is not that NASA wants to do 
things that are more likely to fail; what’s the point in that? The message is that by accepting elevated 
levels of risk, the Agency can accomplish more for the tax-payer dollar, even with increased probability 
of increased mission failures. This can be a difficult message to Congress, the media, and the public. 
Advocates of small, risk-tolerant missions can point out that such a portfolio, even with an increased 
failure rate, gets more done for the money since fewer resources are expended trying to assure no failure. 
The key for the stakeholder community is portfolio management and messaging. NASA is a public 
agency, so unlike other more secretive agencies within the government, everything NASA does is 
intentionally open for all to see, and a public failure is a public failure. This means that a risk-tolerant 
mission portfolio needs to be explained very carefully so that a failure does not crush the portfolio. It 
needs to be clear to all that there is a part of the Agency portfolio where it simply makes sense to take 
risks in order to move technology maturation and capabilities along cost-effectively, and the cost 
associated with trying to assure there no failures is too expensive. The mindset of the stakeholders would 
need to be in alignment with this reality. 


While Class D missions can 
accept the highest levels of 
risk, they actually require the 
most rigorous risk management. 
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In general, NASA has grown a culture which is fairly risk-averse. Much of this comes from some of the 
highly visible failures which involved brave astronauts in harm’s way. Human missions are not Class D 
missions; however, and robotic missions put no human life at risk. In this part of the portfolio, NASA 
needs to be able to do bold, risky things. NASA is supposed to be walking the edge and pushing the 
envelope. This situation is analogous to the contextual differences between flying an F-16 fighter aircraft 
and flying an unmanned drone. One is an incredibly sophisticated, high-performance vehicle which 
carries humans and the other is a less-expensive, non-human vehicle which can accomplish a number of 
meaningful tasks. These two vehicles span mission class from A to D, and they both have a place in the 
broader portfolio. 



Figure 1 5-1 : LCROSS & LRO on the launch pad at Cape Canaveral 

Florida with the Space Shuttle in the background. 


The key is up-front messaging, followed by continued messaging throughout. Once a mission fails (or 
even has a setback), you will not be able to explain the nature and risks associated with a Class D 
portfolio, as explanations will just appear as excuses for the failure. This was very apparent during 
LCROSS’ later on-orbit anomaly, which will be discussed in great detail in a subsequent chapter. In all 
the media interviews I conducted from the beginning of the project, I would note the risk-tolerant 
position of the LCROSS mission and how cost and schedule were driving requirements. That message 
would successfully get out in some of the articles, but by and large, the general public and media really 
had no idea that LCROSS was a different kind of mission. Many viewed it as an F-16 when it was really 
an unmanned drone. LCROSS was compared directly to missions five times its cost. LCROSS was simply 
more likely to fail (few redundant systems, less testing, reduced oversight, etc.), but with such a low price 
tag, more missions like it could successfully fly, and in total, net more benefit for the Agency. 
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There is a place for $1B missions and there is a place for $100M missions. If the portfolio is properly 
designed with appropriate overlapping of low-cost, risk-tolerant mission capabilities, the portfolio can be 
resilient to a certain number of mission failures and still get more accomplished than a single flagship 
mission whose cost is strongly driven by efforts to assure no failure. 

Another facet of messaging: Use of Commercial off the Shelf (COTS) instruments. My concern I had 
early in the inception of the LCROSS mission proved to be spot-on during the flight of LCROSS. As 
noted in earlier chapters, LCROSS chose to largely use COTS instruments as part of our payload suite. 
This proved to be a substantial cost-saver if we could convince ourselves that they were worthy of launch 
and spaceflight. Employing COTS instruments led to our payload budget being an incredibly small 3-4% 
of the overall project budget, when traditionally payloads represent 20% or more of the total mission 
budget. This savings comes from a great reduction in that amount of required custom designs and 
verification testing, which are labor ($) intensive. The trade when using COTS instruments is not only a 
potential increase in risk since the instruments were not originally designed for spaceflight, but also that 
these commercial instruments do not necessarily push the envelope in capability. By definition they are 
mass-produced capability at a good price. The final LCROSS science results speak for themselves. . .“but 
what’s up with the grainy images?!” As I noted earlier, I saw this coming from the beginning; the 
LCROSS science imagery was perfectly suited for our science application (and cost framework), but they 
were rather unimpressive to the media and general public who was accustomed to high-definition 
imagery in everything from their televisions to video games. LCROSS appeared to some to be making a 
40-year leap backwards to capabilities of Apollo-era instruments. 

It was interesting to see the blogosphere react with comments like, “My parents saw images like that 
from the surface of the moon on Apollo. . .what gives?!” “What’s up with NASA that they haven’t moved 
their own technologies forward along with the rest of the world over the past 40 years?” “Where’s the 
High-Def?!” My personal favorite was one blog in which someone commented on the live-feed images 
he saw from the NASA Mission Operations Control Room (MOCR). He noticed a standard power strip 
on the floor and commented, “Did you guys see that power strip in the NASA control room?! That thing 
looks like they could have bought it at a Home Depot!” Now while I agreewe might’ve done a better job 
of hiding mundane things like power strips from the view of the camera, I can assure all that read that 
blog thread, we did buy it at Home Depot!, and saved the taxpayers money in the process! In the end it 
powered some of the peripheral systems that ultimately supported discovering water on the moon. There 
is no shame in that. 

When I read these comments I wished I could have personally addressed each one because the answer 
was so simple. The problem was that NASA had not done a good job expressing the impressive cost and 
schedule feat that LCROSS had accomplished. Our COTS instruments literally came from the beer- 
brewing industry, NASCAR, and carpet recycling applications, and these instruments were being 
compared to billion-dollar investments of the Apollo era of the 1960s. I am personally impressed by the 
fact that we flew common, very low-cost instruments and were able to discover water on the moon. 
LCROSS made a fundamental change in the way humankind views the moon... at the NASA-equivalent 
cost of chump change! 
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Figure 15-2: Daniel Andrews (PM) and John Schreiner (MOM) in the control room following LCROSS 
lunar impact. (The power strip is located on the floor behind Dan.) 


We should have done a better job communicating this facet of the LCROSS mission so that the public 
and stakeholder community had the proper expectations for the mission. I am convinced that that would 
have actually invigorated the public. Not only was NASA doing right by the American taxpayer, but it 
was being extremely clever and resourceful with a mission that was pushing the limits of low-cost. 


Timing of the messaging is of course everything as I noted previously. Delivering the message after the 
fact just doesn’t work. When LCROSS experienced its on-orbit anomaly, I knew that the time to explain 
Class D had passed. Attempts to describe our elevated risk position would simply appear to be posturing 
by the Agency - as if we were trying to prepare the public for the imminent doom of the mission. The 
Agency should have been way out in front with the 


LCROSS messaging, but it was now academic. 
Where it had succeeded in garnering great 
attention about the excitement of the impact, it had 
failed in explaining the real nature of the mission. 


LC ROSS made a fundamental 
change in the way humankind views 
the moon.. .at the NASA-equivalent 
cost of chump change! 
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To further complicate the task of messaging LCROSS, the Agency increasingly wanted to showcase this 
interesting mission to the public, as it was such an exciting event; it was an "explosion, ” afterall! The 
team fully endorsed this idea because exciting opportunities like LCROSS can greatly influence the next 
generations to see how cool math, science, and technology is and how it can be applied. However, as the 
interest in LCROSS grew it only magnified the Agency’s dilemma about the risk tolerance message. 
Given the attention this mission is now receiving, is LCROSS no longer “low priority”, per its NPR 
8705.4 Class D definition? If LCROSS failed as millions around the world watched, wouldn’t it be a 
huge blow for the Agency? 

The answer is to stay in front and manage the message . Since the Agency engages in all mission classes 
(A, B, C, D), it has a portfolio spanning the whole spectrum. The Agency needs to remind the public that 
what it does is inherently risky because of the harshness of both the launch environment and of space; 
and once you launch a spacecraft, you generally can’t get it back to the shop for repair. Having the 
Agency remind the public that it has a “well-diversified” portfolio of missions that span levels of risk 
based on safety and cost is essential. 

One of the more notable disconnects between the Agency and the public related to the impact itself. 
LCROSS ballooned in popularity because of the unconventional nature of this type of mission. We were 
not going to soft-land on the surface of the moon and gently extract samples with a slow-moving 
scraper.... We were going to strike the moon so that we can look through the rubble and dust that kicks- 
up and efficiently (and inexpensively) see what is down in that crater! NASA got the public excited about 
an event which would plausibly be viewable from certain locations on Earth - this was the truth. The 
messaging that was not properly handled with the public was that nobody knew the makeup of the 
bottom of the crater floor. It could be anything from incredibly rocky where the impact force could cause 
a visible flash condition of hitting rock, to talc-like dust which would simply swallow the impactor 
heating-up the local soil and causing more of a vapor cloud of released volatiles than a spectacular 
explosion. There was little way to kn ow in advance, and in fact, which we witnessed is itself a clue to the 
nature of the permanently-shadowed craters on the moon. It all adds up to good science, but when poorly 
communicated to the press results in a great let-down of expectations. 

Further complicating the story with the public was the nature of crater selection. The beauty of LCROSS’ 
mission design was that it could decide which crater it was going to impact well after launch. The reason 
this is attractive is the LCROSS mission scientists can be informed by data received from other missions, 
such as LRO and allow it to select craters which would have the very best scientific return. This is fantastic 
news any way you slice it - except when it is not properly messaged to the public. I still recall a 
conversation I had with the LCROSS Chief Scientist as we had to decide the impact crater near the end of 
the mission. We had a dilemma. We could pick a deeper crater which by all the most recent in formation 
looked very attractive for water-ice - thereby improving our science return, or pick a shallower crater which 
may be less attractive scientifically, but by being shallower, would provide the best Earth viewing of the 
impact ejecta. Science priorities versus public interest? We of course chose to maximize the scientific return 
for LCROSS which meant there was a probability that the ejecta that rose from the impact might not get 
high enough for it to be visible over the local lunar terrain, from Earth. As it tums-out both the nature of the 
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crater floor (talc-like) and the crater depth worked against the ability of humans on Earth to see the ejecta. 
In fact it was even difficult to discern the ejecta from the LCROSS shepherding spacecraft itself until we 
look at thermal imagery was it was very clear (since we could see the local heating from the impact ejecta). 
All of this needed to be more effectively messaged to the media and public. 

I think the broader public inherently understands 
that NASA does risky things and endorses it. If 
NASA did safe, easier things, then it shouldn’t be 
in the business because the commercial sector 
should pick up that work and make money doing 
it. The tax-payers want to know they are getting 
good value for their investment - that their money is not being squandered. For this reason, the Agency 
must be clear that when it takes on riskier missions, or makes decisions regarding science return, it does 
so because from a portfolio perspective. It is making the best investment in either moving technology 
forward or answering fundamental questions about our universe and our place in it. 


The Agency must be clear that 
w hen it takes on riskier missions, 
it does so because from a portfolio 
perspective, it is the best investment. 
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Surely, when LCROSS was selected everything started out beautifully with all the stakeholders and 
peripheral observers wanting nothing but success for LCROSS, right? They most certainly had full 
confidence in the team and gave it wide latitude in how it ran the project, right? No, not so much. While it 
wasn’t a hostile environment, the stakeholders clearly felt some uncertainty and caution. As a team, we 
had few relationships established with the stakeholders, and as a Center, little recent mission history. There 
was clearly some uncertainty about a NASA-ARC team leading the LCROSS mission. Further, one of the 
many clever aspects of the LCROSS mission was its leverage of some designs and even procurements of 
the LRO mission, thereby maximizing the return for the Agency; however this brought NASA-GSFC to 
the LCROSS table. Just a few weeks earlier, NASA-ARC and NASA-GSFC had been in competition for 
the secondary payload slot that NASA-ARC had just won with LCROSS.... So there was plenty of 
circumstantial context that the LCROSS team had to address and overcome in order to be successful. 

As the LCROSS project manager, I was one of the principals on the team in setting the tone of the 
internal and external relationships of the project. These relationships are essential in the success of a 
project and later mission. The PM carries the responsibility of being the face of the project. The PM 
creates the first impression of the team, and with this responsibility comes the need to assure information 
exchange and collaboration are productive and functional. The key is to get team-partner-stakeholder 
alignment through shared goals so that individual team members can make individual decisions that align 
with the broader vision. That’s where we needed to be - but was not where we started. 

“Can we trust NASA-ARC with a mission?” NASA-ARC hadn’t managed a flight project to launch in 
a long time, and other projects in recent times had had some difficulties. The most recent NASA-ARC 
spaceflight mission was Lunar Prospector (LP), flown in the late 1990s. While considered a pathfinder 
for low-cost approaches, the NASA-ARC scope on LP was much smaller than that of LCROSS. To go 
back even further in history, you land in the 1970s to find an NASA-ARC-led flight project. NASA-ARC 
had something to prove with LCROSS. 

“The LCROSS NASA team is green and inexperienced.” This was mostly true. While the NASA- 
ARC team was made up of very experienced engineers and scientists, they had mostly worked on non- 
flight activities. Some did have experience with flight payloads or support roles to the International 
Space Station (ISS), Shuttle, or Mars Exploration Rover (MER), but these were generally not lead 
activities. In the end, LCROSS proved to be an excellent opportunity for the LCROSS team of 
enthusiastic, talented individuals to experience leadership roles on a challenging project - something 
increasingly needed in the Agency as previous generations retire. 

“Center rivalries will be a problem.” When LCROSS was selected, there was some heat that naturally 
surfaced. While NASA likes the model of operating as “One NASA,” it establishes competitions for 
important work which pits the parts of NASA against each other. So some of the heat came from natural 
winners and losers that come from a selection, but there was further intra-center conflict associated with 
the LCROSS Class D approaches being so unconventional as compared to traditional NASA methods. 
These approaches could represent a potential threat to others’ ways of doing business. Each Center has 
different cultures and approaches, adopted from the Agency guidelines. How NASA-ARC would execute 
LCROSS was not clear to most. 
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Figure 16-1: LCROSS/LRO packed in the launch fairing, moving out to 

the launch pad for integration with the launch vehicle. 


“Can LRO and LCROSS work together?” Frankly, the LRO project team had every reason to not be so 
happy about an LCROSS secondary mission sharing their ride. LRO was past their PDR review, heading 
toward their CDR and the Agency was not inserting another mission under them in the launch stack. This 
co-manifesting had the makings of an arranged 
marriage at best. Can two different missions from two 
different centers not only get along, but collaborate? 

“Can Northrop-Grumman be agile and cost- 

effective?” Few questioned NG’s ability to produce a 
fine spacecraft, but few also believed that their culture 
would support producing a spacecraft in a cost-effective and timely manner. How can the demands of 
LCROSS be supported by the aerospace behemoth, Northrop-Grumman, in a cost-capped fashion? NG 
was motivated to perform on exactly this type of work, but it would require changing their approaches 
and tailoring their internal methods. 

The LCROSS project faced a significant challenge. We needed a trusting, healthy relationship with the 
stakeholders and with all our partner suppliers, whether supplying hardware, software, or influence. The 
key to the successful relationships LCROSS established was openness and trust. 


The LRO project team had every 
reason to not be so happy 
about an LCROSS secondary 
mission sharing their ride. 
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Center-to-Center Trust: The first trust issue 
was a NASA Center-to-Center problem. 
LCROSS was making use of the LRO avionics 
designs as a basis for its own avionics, as well as 
looking to join some existing LRO procurements 
to save the lead time constraints of starting fresh 
procurements. These approaches were clearly to 
the advantage of the broader Agency as it was 
benefitting from non-recurring cost savings. For 
this to be effective in reality, LCROSS needed to 
have a good working relationship with the LRO 
team and more broadly with NASA-GSFC. 
Trust issues were apparent from the beginning. I 
personally saw trust issues surfacing from 
NASA-ARC and NASA-GSFC, mostly at the 
executive management level. NASA-ARC and 
NASA-GSFC were jockeying, and that is 
usually not friendly to an already stressed 
project schedule. The bright light, however, was 
with the LRO project manager, Craig Tooley. 
Over the course of both projects he and I 
established very open communication and 
always tried to take the long view for the 
Agency. This was not just about LRO or 
LCROSS, but about the success of both of them. 
I needed to be very careful in not causing 
negative impacts on LRO’s development and 
risk position and LRO needed to provide 
LCROSS needed data from which to leverage 
with its own development. In the end the benefit 
was to both projects, not just LCROSS. 



Figure 16-2: LCROSS/LRO stacked atop the Atlas-V 
launch vehicle at the Vertical Integration Facility (VIF), 
Space Launch Complex 41, Cape Canaveral. 


Once the Centers settled back and allowed the projects to run, things proceeded much more smoothly. We 
were mutually respectful of each other’s needs and demonstrated that in our interactions. Eventually the 
LCROSS and LRO teams agreed that LCROSS would place a liaison down in Maryland to live in the 
same hall as the LRO team, to get full engagement with their activities and to facilitate communications 
between the projects. This ended up being very helpful to both projects. The liaison not only got answers 
for LCROSS on matters of shared interest, but also promoted the exchange of information and data that 

. , . . , . LCROSS discovered about hardware 

The key to the successful relationships and software that LR0 

was using, 

LCROSS established was openness and trust, saving them time, i found it very 
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gratifying to hear that the LRO team essentially adopted the LCROSS liaison as one of their own, further 
engendering camaraderie between the two projects. An effective liaison cannot simply be someone who is 
willing to collocate. It takes a certain personality to pull off the role: someone who will first, not be a 
burden, second, be a good listener, and last, bring actual value to the team where they are stationed which 
means being technically competent. 


Another angle on Center-to-Center stress involved the move I htf project’s greatest 

of the program office which oversaw LRO and LCROSS, $tren°th is its tranSDareilCV 
from NASA-ARC to NASA-Marshall Space Flight Center 

(NASA-MSFC). It was one thing for the small LCROSS project team to report to a local program office at 
NASA-ARC, but moving the program office to NASA-MSFC could require a lot of travel and entailed the 
addition of yet another NASA culture and stakeholders to the mix. At one point, a senior leader told me, 
“They’re snakes - do not trust anything they tell you,” revealing the level of hostility I had in front of me. 
I somehow had to bridge the gap between all these parties and agendas in the best interest of the project. I 
remember expressing to my local management that I understood there was some history, “. . .but in the end, 
I need to work with all these people. . .and in a productive way to move this project forward.” We agreed 
on this point. 


The approach I took was openness and candor, which 
suited the way I work anyhow. If we could show each of 
the stakeholders that we were candid and timely with 
information, both good and bad, we’d earn their respect, 
even if those stakeholders were at odds with each other. 
As this took hold over time, we later heard comments 
from the stakeholders like, “The project’s greatest 
strength is its transparency,” and this from our mission 
manager in the NASA-MSFC program office no less. 
This trust was cemented when we established a weekly 
telecon with the program office, in which we’d share our 
status frankly. We even opened the telecon up to HQ 
program executive attendance, which was uncommon. 
This meeting became one-stop shopping for all 
stakeholders who wanted to hear the latest with the 
project and its progress and to get questions answered. 
We also set up a weekly telecon with our sister mission, 
LRO. This did not happen from the get-go because of 
some of the early political issues, but once started, it 
became a very productive means for exchanging needed 
infonnation. The relationship between the LCROSS PM 
and the LRO PM was always solid, even if there were 
complications between our centers. I think we found 



Figure 16-3: LRO/LCROSS heading out to 
the launch pad in Cape Canaveral, FL. 
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common purpose in the tough challenges we shared going forward. That LCROSS/LRO weekly telecon 
was a great venue both to coordinate and to share status between the two missions. To offset the time that 
some of the LRO staff spent coordinating and answering LCROSS questions, the program office 
provided some small funding for this activity to LRO. This exchange of information proved to be to 
LRO’s benefit as well, because later in the project development, the LCROSS team made discoveries that 
proved to be very helpful to LRO. 


Project-to-Stakeholder Trust: I refer to a stakeholder as any party that has a vested interest in seeing a 
project (in this case, LCROSS) succeed. This can include other centers (program office), HQ (mission 
directorate), NASA-ARC (performing center), and even the contractor (Northrop-Grumman). One of the 
first, broad tests of our management of the stakeholder community came at the LCROSS Systems 
Requirements Review (SRR). It was our first big “gate review” in the development of LCROSS and just 
about all the stakeholders were present... and boy were they were scrutinizing us. It was also the first 
time we had to rein in the degree of programmatic oversight. At one point, more than thirty people from 
the program office and NASA-HQ were planning to attend the SRR, which was comparable to the total 
number of people staffing the project at NASA-ARC! NASA-ARC management brought this matter to 
the mission directorate Associate Administrator. In response, Doc Horowitz stipulated that the program 
office could bring “no more than one person per $10M of project value.” This of course further illustrates 
the tension between the stakeholders (NASA-Headquarters and the program office) that the project had 
to navigate, since we had to answer to all of them. 



Figure 16-4: Lift-off of LRO/LCROSS on June 18, 2009. 
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As I reflect back on that SRR, it was 
somewhat amusing because ESMD 
had clearly brought in a ringer, or as 
we had later heard him called, a 
“ hired gun ” from the Applied 
Physics Lab (APL) to challenge us. 

It was apparent he wasn’t out to s ink 
us, but to make a critical assessment 
of our work and progress. As we 
presented, he and the new Lunar 
Precursor and Robotic Program 
(LPRP) program manager from 
NASA-MSFC asked fair but 
penetrating questions, clearly trying 
to test our openness and candor. I 
suspect our relative inexperience 
was apparent, but I also believe that 
the frankness and genuineness of 
our presentation were even more 
apparent. We were honest and 
candid, and fielded questions in a 
straightforward manner. We even 
got into a discussion of Faster, 

Better, Cheaper and how I saw our 
project being different. In the end, it 
was a satisfactory SRR, but more 
importantly it was a trust-building 
meeting. I was later able to confirm 
that the APL ringer was indeed 

brought in by the mission directorate specifically for that purpose - to see what was really there on the team. 
I think that review bought us more goodwill and autonomy with the stakeholders than nearly anything 
else. . .with maybe the exception of the PDR. 



Figure 16-5: Actual alligator warning sign at Cape Canaveral. 


For any mission, the Preliminary Design Review (PDR) is pivotal, as it is the first opportunity for 
stakeholders to assess the team’s technical prowess against the backdrop of the programmatics (cost and 
schedule). Not as obvious, but equally important, is the stakeholders’ assessment of the team’s operation; 
that is, how well it functions and LCROSS provided an unintended clinic on that very topic. As we kicked 
off our presentations on the first of two days, we started out great. The first two presentations went quite 
well, with good engagement by the audience in the speaker’s material. However, the third presenter, who 
represented an important facet of the LCROSS team, got into some trouble. The material was thin, and the 
verbal presentation seemed to worsen it. You could feel it in the room - concern and even fear. 
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After the presentation, the Ames Chief Engineer even slipped me a note on a scrap of paper which read, 
“After that, you’re going to have a delta review,” confirming what I was sensing in the room. I was being 
told we would not pass this review and a “delta PDR” would inevitably be required. I realized the cloud 
in the room would color the whole rest of the review, as we were only on the third presentation on the 
first of two days. After the speaker finished and came back to his seat, I asked him how he felt it went, 
and he could tell it could have gone better. I told him I wanted to set up a splinter session for that very 
day to give him and me the chance to speak with the stakeholders, round-table style, and attempt to put to 
bed any of their concerns or fears. He agreed, and while the presentations were still marching along, I 
arranged for a meeting room with our NG hosts, and then walked around the room, personally inviting 
each key stakeholder to the splinter meeting for that afternoon. 


When we got together, it was awkward. I wasn’t sure where to begin, but started by saying that I thought 
our content in this area could use some clarification and wanted to get the stakeholders all together to 
enable them to share any questions or concerns they may have had with the speaker, with me, and with 
the project. We then went around the room, one-by-one, giving each stakeholder the floor to share their 
concerns and to allow the speaker to address them individually. 


I’ve never experienced such a level of 
openness on a spacecraft development. 


This splinter session proved to be invaluable, 
as it not only gave the speaker the chance to 
save his reputation, but it also demonstrated to 
the stakeholder community that the LCROSS 

project wanted to do right by them and would go the extra mile to make sure they were on board. At the 
conclusion of the LCROSS PDR, and even days later, I received comments, through emails and calls from 
stakeholders, telling me that calling that meeting was exactly the right medicine and flipped a negative 
situation into a strong positive. It affirmed in their minds that this team was playing it straight and was 
receptive to stakeholder commentary and experience. Reviews like these and the reviews that followed 
continued to earn points for the project. The Office of Chief Engineer at NASA-HQ would later comment, 
“I’ve never experienced such a level of openness on a spacecraft development.” 


NASA (Ames)-to-Contractor (NG) Trust: When LCROSS was still in the proposal phase, we were 
already forming a relationship with NG. It was a little rocky at first, as both sides were feeling each other 
out; the long hours and tough workload of proposal writing provided focus that could either fracture a team 
or weld it together in the heat of battle. I was the proposal manager for LCROSS in the final round of the 
proposals and my counterpart at NG was a very experienced proposal person - a great front-end resource. 


As the LCROSS proposal team went through the permutations of what this mission would do, how it 
would do it, and how the work would be divided, a couple different times “the wheels came off the cart.” 
This was a phrase my NG counterpart would share when something wasn’t closing on feasibility. The 
toughest one with NG occurred when they were really stretching what their corporate structure would 
tolerate with respect to the size of their team. A small mission like LCROSS required a very small NG 
team and this was making NG senior management very nervous. In the end, we worked through those 
advocacies, and with each one forged a stronger and stronger relationship. These relationships always 
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paid forward benefits. When LCROSS was selected, a NG project manager was brought to the table and 
was coached in advance about what a great, hard-working team was up at NASA-ARC and how well 
they worked with their NG counterparts during the proposal. This sowed seeds for starting the next 
relationships with less time in the “dating phase.” 

LCROSS was to work with two NG project managers over the course of the project and mission. Steve 
Carman managed the spacecraft build through design and production, and Craig Elder managed the 
spacecraft through delivery, launch, and mission operations. We had an extraordinary relationship 
throughout. This is not to say we didn’t have to truly negotiate on the cost of contract mods and the rest. 
Sometimes we flew down to NG to specifically negotiate the costs associated with a mod and it got a 
little warm in the room. During one negotiation, the NG SE stated “Wow, I’ve never seen anyone 
question $60 in one of these negotiations.” It wasn’t about being cheap, but about being cost conscious, 
regardless of which side of the table we sat. The discussion never strayed from being professional and 
fair, however. We both needed this to work-out and we both knew it. We collectively tried to be honest 
about which changes were truly NG’s ownership, which ones were truly NASA’s ownership, and in 
which ones both parties should share the responsibility. Even these times helped to strengthen the 
relationship and made future interactions that much more easy and productive. I cannot emphasize 
enough how important this is compared to the opposite situation, which can kill a project. The moment 
that each party is in it to just to meet their own needs at the expense of the others, is the moment the 
project is doomed to overrun and maybe collapse. 



Figure 16-6: Dan Andrews (PM) and John Marmie (dPM) working from the 
residence in Florida in preparation for launch. 
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The mission that some stakeholders 
believed was doomed to fail from 
the start is suddenly looking 
like it might actually make it. 


So as this trust between the stakeholders continues 
to build and you move through the project, an 
interesting thing starts to happen: people outside 
the stakeholder community start to take notice. 

The mission that some stakeholders believed was 
doomed to fail from the start is suddenly looking 

like it might actually make it. We started to see this after a successful Preliminary Design Review (PDR), 
and most definitely after the Critical Design Review (CDR). I still remember the day that the NASA- 
ARC Center Director told me he was in a conversation with the NASA Associate Administrator of the 
Science Mission Directorate (SMD) - a sister directorate to the one where LRO and LCROSS were 
housed. He said that the Associate Administrator for SMD wanted to run missions in his directorate the 
way LCROSS was being run. The perception coin was flipping in the minds of critical external 
stakeholders. We knew we had moved from being a potential Agency liability to being a true opportunity. 


Confidence, trust and general tone come from openness and finding common purpose, whether within a 
team, or a team and its partners, or a team and its stakeholders. It’s OK that an industrial partner is 
ultimately driven by “the bottom line” as long as you can establish common purpose that helps them 
achieve that outcome. It’s OK for the high-level stakeholders to have political aspirations and 
motivations for success as long as that common purpose is in synchronicity with the needs of the project. 
The Agency’s needs are different from the commercial contractor’s needs, which are different than the 
needs of the program office. But in satisfying those individual organizational needs, we have to find 
common purpose. For LCROSS, NASA-ARC was looking to get back into the spaceflight game, while 
Northrop-Grumman was looking to carve out a new, responsive means for small spacecraft development. 
The key was for both organizations to walk the road together. A “team” is “a group of interdependent 
individuals with strongly focused, shared performance objectives .” 1 


A strong team is also led by a strong leader. This does not mean a dominating leader. I think hardly 
anyone on the LCROSS team would state that I was a dominating micro-manager - quite the contrary, I 
depended on the team leads to manage their development as if they were project managers. However, 
decision-making by the team must be clear and definitive. We would call project review boards (PRBs) 
when a critical decision was required. In the end, after all the points of view were expressed, there had to 
be a definitive decision. 


This is especially critical for small, cost-capped teams 
because there is little time for deep reflection on ways 
forward. The team expects the project manager to make a 
call after all the opinions are put forward. Even where there 
is an attempt at consensus, in the end, the team wants to 
kn ow there will be a decision. That is the role of the project manager, who must then enforce that 
decision with the team as they move out. “The credibility of the team is directly correlated to the 
credibility of the team leader .” 1 


The credibility of the team 
is directly correlated to the 
credibility of the team leader. 
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Figure 16-7: LCROSS Team Apparel - One of the Facets of Teambuilding. 
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So managing the project tone is critical to establishing good communications, engendering trust, and 
creating an environment where people are the most important asset of the mission. Small teams rely to an 
extraordinary degree on individuals and their performance in the context of the team. As noted in Spear’s 
FBC Report, “This requires unprecedented teaming and open, candid communications. No one person 
has the answer. It takes a lot of debate and evolution of ideas to get there. It’s about getting them to align 
their direction, their ego vectors, in the same direction as the project .” 2 


End Notes 

1. Bingham, N., “Teams: Building and Breaking,” Critical Thinking Seminar, January, 2009, NASA, Ames Research Center. 

2. Spear, Tony, “NASA FBC Task Final Report,” NASA, 2000. 
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A good part of the success of the LCROSS mission is 
attributable to the LCROSS stakeholders. This is not to 
diminish the extraordinary LCROSS team, but for the 
most part, the LCROSS stakeholders were steady 
throughout the entire project, keeping funding consistent 
and requirements steady. This is unfortunately not very 
common within the Agency for a number of reasons, and 
yet is so important to enabling a project to be successful. 

A project always exists within the context of its 
customers, and if they are not in alignment with what is 
required for project success, and hold to commitments 
made, the project will be strained if not doomed. In the 
case of LCROSS, The NASA-HQ mission directorate 
and the program office were very much in alignment 
with trying new, lightweight approaches that would 
enable LCROSS to be successful.... However it was 
entirely unclear what those approaches would look like and the stakeholders were not necessarily on the 
same page at an individual level. 


Figure 17-1: Combined LCROSS/LRO/Atlas 
Mission Logo, designed by Roger Amo. 



It is important to understand that the stakeholders had different reasons for wanting to see LCROSS 
succeed. Even with shared goals, they had different sensibilities about how to get to the end successfully. 
Everyone understood that the Class D construct enabled lightweight design and test approaches, but it 
was unclear what this specifically meant with respect to the Agency’s procedural requirements. We 
needed to define an efficient, achievable methodology for the project if we were going to be successful. 


One simple example is the requirement to create a 

project plan. The project plan is the roadmap for The COSt of any data products 

executing the mission, and once signed, constitutes an should be COIUIU disunite with 

agreement with the program office, given the program the “value” of the data product, 
manager signs it. The project plan constitutes a means 

for managing expectations between the performing team (LCROSS) and its principal stakeholder (the 
program office). While the NASA project plan contains many details about the plans of the project, it 
doesn’t actually capture the constraints under which the project has to operate. The team and I spent a fair 
amount of time developing the project plan during formulation of the project, documenting content from the 
stipulated NASA policy requirements, and it did help organize our thoughts and plans about how we 
planned to manage the project; however, over the course of the project, it was rarely looked at because it 
was filled with template-required content that was of little value to the project - more for the benefit of the 
stakeholder. The project plan was put on a shelf, being pulled-out only a couple times over the course of the 
project. Shouldn’t this document be the guiding document of agreements between the project and its 
stakeholders? I do know this: the labor cost for creating the project plan was much greater than its value at 
the time of LCROSS, and for a low-cost, challenging project, the cost of any data product should be 


130 


Chapter 17 — Managing the Stakeholders 


commensurate with its “value.” Since the time of LCROSS I am seeing changes in the Agency policies 
which are moving in this very direction with updated agreements being appended to the project plan to 
make sure all parties are on the same page. The project plan should be a living document and not a dusty 
origination document which is not used. There is no shame in updating the project plan if the constraints 
under which it exists are changing. In this way, the project is actually protected from external changes 
because they would immediately fall out of congruency with the stakeholder-approved plan. Of course a 
similar obligation is on the project team to honor its commitments which are hard-coded in the plan as well. 


Another stakeholder management approach was to 
provide complete transparency in project execution. 
This isn’t to say that we relinquished our ability to 
make decisions and strategize approaches as a team, 
but we allowed stakeholders visibility and interaction 
with the team at an unprecedented level. We figured th 
our statusing, the more efficient the project’s flow and 


We allowed stakeholders 
visibility and interaction with the 
team at an unprecedented level. 

e less translation and reiteration we had to do with 
the stronger the buy-in by the stakeholders. 


This transparency began with the established project-stakeholder interactions. If not handled properly, 
this could be labor-intensive and even counterproductive if not providing actual value to the project. For 
this reason reporting had to be continually watched and steered as necessary. The following list describes 
the established management meetings held with the stakeholders: 


♦ Weekly program office telecon. This meeting was owned and managed by the program office, 
with the project being the principal briefer on progress and issues. The telecon’s agenda included 
first a verbal status from the project manager, then from the other participants, and generally lasted 
60-90 minutes. The project’s Level- 1 stakeholder (NASA-FIQ ESMD) was also in regular 
attendance in order to be efficient with the reporting. Further, we had the Launch Services Program 
(LSP) in attendance (as appropriate) in order to elevate and address topics related to the launch, 
including LCROSS/LRO co-manifesting topics. When we had important decisions to make, this 
meeting was also attended by representatives from the NASA Office of Chief Engineer (OCE), 
NASA Office of Safety and Mission Assurance (OSMA) and other stakeholders who just wanted to 
hear the latest from the project. The one-stop-shopping nature of this stakeholder communication 
means meant it was well-attended and efficient. 


♦ Weekly Launch Services Program telecon. This meeting was owned by the program office as 
well, but the principal briefer was the Kennedy Space Center (KSC) integration manager for LRO/ 
LCROSS in order to enable the project and its stakeholders to understand any issue or topics that 
required action or discussion. This was a good meeting, for sometimes unintended reasons. The 
very nature of the LCROSS mission was challenging for the project team, but was also very 
challenging for the launch services program and the launch service provider. We were running at a 
pace that was much faster than they were used to with typical NASA missions. This meeting 
enabled the LCROSS team to communicate our needs from the LSP on a timeline and this almost 
always caused some stress since products were not typically available until much later in the flow. 
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This meeting made it publically clear that we were lacking information needed to support our 
schedule and that could get high-level attention from the Agency with the hopes of remediation. 
With LRO project representatives attending as well when we got close to launch, the meeting was 
especially lively - especially when the LCROSS launch date was being moved by forces outside 
the project’s control, thereby requiring large scale replanning for the small LCROSS team. 


♦ 


Weekly meeting with NASA- ARC 
Center Director and Staff. This 
was an NASA- ARC internal 
meeting in which the project (with 
whom the Center is responsible for 
the technical performance) briefs 
the Center Director of NASA-ARC 
as well as his executives. This 
meeting hosted a “five-minute tag- 
up” by all the Center projects, 
wherein they were directed to report 
only status and problems - no “atta- 
boys.” The idea was to give the 
Center Director the chance to 
address issues before others were 
briefed on the same, with the goal of 
having the Center Director’s 
influence to help remediate issues. 

This was critically important for 
NASA-ARC-internal issues, where 
Center institutional performance 
issues could be elevated, but it was also useful in directing pressure to certain vendors who needed 
some “encouragement” to honor existing commitments. The Center Director of a NASA Center 
carried weight in negotiations with both vendors and even other NASA organizations and this 
meeting was a means for explaining the assistance required. 


Figure 17-2: LCROSS Logo on the Side of the Centaur 
Rocket, Making this Feel Very Real. 



♦ Biweekly Ames Chief Engineer (ACE)/LCROSS PM meeting. While the PM does not have a 
normal, required reporting path through the Chief Engineer’s Office, the ACE used this venue to 
understand the PM’s thinking on matters of a technical nature. It is equally important for the PM to 
understand the Chief Engineer’s concerns as it is for the Chief Engineer to understand the 
programmatic trades the PM must make on technical matters. This meeting could become 
contentious when the programmatic decisions were not in direct alignment with the ACE’s view on a 
technical matter. That’s not a bad thing, though, as it enabled hearing from both sides in a small 
setting that fostered advocacy. In the end, I think this served both parties well. 
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♦ Biweekly Ames Chief 
Engineer/LCROSS PSE 
meeting. This is the one 
meeting in which the PM 
was not a participant, by 
design. It was a venue for 

the PSE to elevate issues with the technical decisions of the project, free of PM influence. It 
preserves the technical authority chain, and as noted earlier, was never a problem on LCROSS 
because of the close, symbiotic relationship between the PM and PSE. 


It is equally important for the PM to understand 
the Thief Engineer's concerns as it is for the 
Chief Engineer to understand the programmatic 
trades the PM must make on technical matters. 


♦ Monthly NASA-ARC Ames Center Management Council (ACMC) briefing by PM. This was 
the PM’s big monthly presentation to Center management, which covered soup-to-nuts on the 
project: financials, risk, schedule, technical performance, issues, accomplishments, media news, 
etc. It was also the setting to show off the project progress with pictures or videos of project 
activity. In the interests of transparency, this meeting was also open to stakeholders (program office 
and ESMD), who attended by phone and even in person on occasion. LCROSS was able to “kill 
two [reporting] birds with one stone”. Our monthly Center briefing and monthly program office 
briefing became a single event, saving the effort of creating and briefing two presentations. This 
consolidation worked because the needs of the program office and the needs of the Center usually 
overlapped. The program office required additional detail that the Center didn’t need, so I 
negotiated a set of charts that answered the program office’s needs, and put them in the ACMC 
backup charts for program office use. 

As with any contract between parties, it is 
best to clearly establish each party’s 
expectations at inception, to minimize the risk 
of misunderstanding. LCROSS established 
clear expectations with its stakeholders 
relating to understanding the mission and 
mission development. This included establishing the types of reviews and ownership therein for this fast- 
paced project. The program office and LCROSS Independent Review Board (IRB) worked with the 
project to determine the terms of reference (TOR) for the review. This is where the entry and exit criteria 
are specifically designed for the reviews, based on and tailored from the NASA procedural requirements. 
The stakeholders understood that this process would need to be stream-lined in order for LCROSS to be 
ready to support LRO’s launch date. This process, while I believe it could have been even more 
streamlined, contributed to the project’s ability to complete SRR, PDR, Confirmation and CDR in less 
than a year from Authority To Proceed (ATP). 


As with any contract between parties, 
it is best to clearly establish each 
party's expectations at inception, to 
minimize the risk of misunderstanding. 
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Figure 17-3: Daniel Andrews (PM) and Tom Luzod (LVM) at the NASA Cape Canaveral Atlas Launch 
control room for LCROSS during “Wet Dress Rehearsal,” where the launch vehicle is goes through 
a dry-run of launch day, including actually fueling and delivering the S/C to the launch pad. 


Sometimes managing the stakeholders means steering through organizational changes. As alluded-to 
earlier in this chapter, LCROSS and LRO were initially part of the Robotic Lunar Exploration Program 
(RLEP) managed by NASA-ARC. In May 2006, the RLEP program office was closed and a new program 
office, the Lunar Precursor Robotic Program (LPRP) was fonned and assigned to NASA Marshall 
Spaceflight Center (MSFC). This ownership change had the potential of substantially altering the way the 
project was expected to perfonn, given that a new people and Center culture being introduced into the 
LCROSS reporting chain. Eventually everything worked out OK, but there were some perturbations. One 
immediate change was a new stakeholder requirements layer being inserted between the Agency level 
requirements and the project level requirements. The new program manager assured me that this would be 
transparent to the project, but that wasn’t really realistic. The project would no longer answer simply to a 
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set of “Level 1” requirements, but now has a set of “Level-2” program office requirements as well. The 
project’s requirements traceability had to be updated to properly flow down to the project requirements, 
and there were some new Level-2 ’s introduced as well. After a couple weeks of work, this settled out and 
the project had a stable set of Level-2 ’s from which our project Level-3 requirements could flow. 

The principal management stakeholders with LCROSS were the program office mission manager (MM) 
and the ESMD program executive (PE). 

♦ The role of the PE is to monitor and report the progress and status of the project within the ESMD 
at NASA-HQ. The PE performs the same monitoring and status reporting functions as the program 
office mission manager, but for the NASA-HQ stakeholders. LCROSS had three PEs over the 
course of the mission with some very different personalities. Traditionally the PE role sits at a fairly 
high level of abstraction over the project and does not get involved in the low-level details of the 
programmatic and technical execution. The longest-running LCROSS PE, however, was a 
technically-savvy individual who knew a great deal about spacecraft systems and was very 
involved in the technical evaluation of the project at low levels. In general, this was a good thing 
because he was an independent technical eye on the evaluations and decisions of the project, while 
understanding that the mission should not be taking technical direction from NASA-HQ. A fine line 
to be sure, which played to the benefit of the LCROSS project. 

♦ The role of the program office MM also was to monitor and report the progress of the LCROSS 
mission, but to the program office, which had first-line programmatic authority over the project. 
The MM made sure that all the programmatic requirements were being satisfied. LCROSS had two 
MMs over the course of the project and both were very helpful as a sounding board and an 
influencing role supporting the program manager’s decision-making. 


♦ While the PE and the MM offices both worked with the best interests of the project in mind, there 
was occasional friction between the two, which put the project in awkward positions. I think the root 
of most of the conflict here was the overlapping of their roles. For example, when LCROSS held its 
weekly telecon with the stakeholders and both the MM and the PE were in attendance, the MM’s 
decision-making was immediately under evaluation by the PE, who programmatically sat above the 
program office, potentially representing a 


challenge to the MM’s authority. I think 
everyone involved had considerable respect 
for the roles and it worked out, but if the 
Agency is looking for ways to improve 
efficiency and lower cost, addressing this 
redundancy would not only save labor 
associated with these two roles, but would 
also simplify reporting for the PM. 


LC ROSS found itself in situations 
where the Center wanted the project 
to go one way, and the program 
office Mission Manager and mission 
directorate Program Executive 
wanted to go another way. 
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♦ Technical authority for LCROSS was defined by NPR 7120.5, and given our designation as a 
Category III, Class D project, NASA-ARC was the first line of reporting for technical authority. This 
was a good arrangement for the project because it created a local stakeholder with whom to seek 
assistance on matters important to the project. Since many of the Project staff came from the same 
Center, the Center was motivated to assist where it could. Of course with this opportunity comes 
complexity within the stakeholder community. LCROSS found itself in situations wherein the 
Center wanted the project to go one way, and the program office Mission Manager and mission 
directorate Program Executive wanted to go another way. This represented the natural tension 
between the programmatic reporting line (program office and ESMD) and the technical reporting 
line (Center and Chief Engineer). This tension and the reporting path differences are intentional, as 
the Agency wishes to assure the mission from different points of view. It is, however, not especially 
efficient for the small, cost-sensitive project. For Class D projects, I would argue that the reporting 
chain should also be greatly simplified to help the project be successful. 


I have always felt that if 
LCROSS had failed, it 
would have been the last 
Class D mission. 


That LCROSS Class D designation became a topic that really 
challenged these stakeholder relationships and arguably created 
some of the more difficult interactions over the course of the 
project. As noted previously, the Class D designation is defined in 
an NPR, but somewhat vaguely, leaving a lot of room for 
interpretation. Ultimately, safety is the only hard requirement and 

all forms of testing are negotiable with the Class D designation... which directly challenges the comfort 
levels of all involved - my biggest frustration of the Class D designation. Support of risk-tolerant missions 
comes in many forms and interpretations. Some believe this classification simply relates to technical 
approaches, while others believe it extends to programmatic perfonnance as well. To some, Class D should 
be able to commensurately reduce the levels of reporting because Class D mission are not urgent, thereby 
reducing the programmatic burden on the team... yet there is no specific relief provided by the policy 
requirements. Whether personal opinion or organizational culture, the Class D mission is left to navigate 
these uncertain waters through many negotiations with the stakeholders. Because the stakeholders were 
challenged with their interpretations of this risk classification, I have always felt that if LCROSS had failed, 
it would have been the last Class D mission. LCROSS was anointed a Class D designation to enable it to 
have some hope to be viable within its tough programmatic constraints, but the Agency was not unifonnly 
comfortable with what that meant. An LCROSS failure could have been easily attributed to any number of 
decisions associated with how it operated, specifically because it was so different. The problem is the 
alternative is for only the most conservative approaches to be taken leading to great expenditures and a 
general risk intolerance which I expensive - something NASA kn ows how to do well. 


A project cannot be forced to fend for itself when it comes to tailoring the policies and approaches of the 
Agency. This defense takes a surprising amount of effort for the project. While it sounds comforting for 
the stakeholders to request the project to define for itself how it should operate, stakeholder weigh-in can 
lead to a bring me a rock activity, consuming untold resources. Class D activities should be considered 
innocent until proven guilty by the external community. That is, instead of requiring traditional 
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validation performance hoops, it should be 
assumed that the designs and concepts put 
forth are valid until the external review 
community makes an assessment of risk. 
Instead of having to conduct all the normal 
tests of upper-class missions, determine 


Despite the Class D designation, enough 
money has been invested, and enough 
publicity presented, that the Agency 
needed LCROSS to be successful. 
This is the problem of Class inflation. 


specifically what is needed to verifying the 

particular needs of the Class D mission... and nothing more. As an example, LCROSS spacecraft 
Thermal- Vacuum (TVAC) testing consisted of merely 1.5 cycles, from hot, to cold, to hot, and then back 
to ambient. Typical missions will go through many-more cycles to reveal any latent defects in 
workmanship. LCROSS relied on the fact that box-level testing already validated the performance of the 
individual subsystems on the S/C and there are citations in the literature which show 90% of 
workmanship defects show-up in the first cycle.... These are the types of trades a Class D mission must 
make in balancing the buckets and it is important that the stakeholders support a culture of the same. 


In the end, LCROSS did make excellent progress in defining a Class D mission approach. However, no 
matter how well you do, as you get close to launch the expectations for the mission grows, regardless of 
its original risk classification. The Agency understandably needs this mission to go... to be successful. 
Despite the Class D designation, enough money has been invested, and enough publicity presented, that 
the Agency needed LCROSS to be successful. This is the problem of Class inflation. 


LCROSS did experience a number of shining moments with the stakeholders, which showed that the 
construct can work. Remember that decision we made to go back into two of the LCROSS avionics boxes, 
already installed and TVAC-tested on the S/C, to check for that potentially faulty capacitor that could take 
out the mission? Once our decision propagated through the Agency, I got word that the Associate 
Administrator for ESMD wanted more information on this decision. I remember thinking, is this level of 
interest, Class D? I later was heartened to hear that the AA merely wanted to check that the technical 
authority chain understood the project’s decision process, was engaging on the topic, and that the PE, MM, 
and program office were doing their jobs... a good call. The Associate Administrator didn’t want to 
oversee the decision himself - he just wanted to make sure the organizational machinery was working, and 
did not require my defending our decision to him specifically. 

With the LCROSS mission complete and successful, I’ve looked back on the project and its context with 
its stakeholders, and am generally pleased with how it played out. ESMD was a new directorate when 
LCROSS began, and as I’ve had opportunities to interact with other directorates within the Agency, I am 
better able to see just how bold Scott “Doc” Horowitz’s ESMD was. This message needs to continue to 
be explored if the Agency wishes to get more done with less. Failure is always a potential outcome when 
you are doing challenging things, but cost and schedule requirements are equally challenging the 
perceptions of this Agency and its ability to meet its commitments. Managing the stakeholder messaging 
is essential to getting the Agency to the right cost/risk balance. 
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Chapter 18 — Managing the Middle 


The role of the project manager is to manage the team . . 

and their activities (downward focus), but also to * lOUgntSt M t ll«i 1 10 IS " HU 
manage the stakeholders and their needs (upward focus). down-Utld-in responsibilities are 
These are the down-and-in and up-and-out at Oilils with the Up-und-OUt. 
responsibilities of the PM, which can lead to some 

difficult situations when you find yourself caught in the middle. Over the course of the LCROSS project 
and mission, I encountered a number of times in which I had to manage differing stakeholder perceptions of 
the right way to move forward, or to address differing stakeholder goals: up-and-out problems for the 
project manager. Similarly, the team needs a decision on a way forward and I had to assess the situation, 
hear the trade-offs and make a down-and-in decision. The toughest scenario is when down-and-in 
responsibilities are at odds with the up-and-out, as with the LCROSS on-orbit propellant anomaly, and was 
the case where the PM is forced to manage the middle. 


LCROSS 20090629 05:23 UT 
P.Mortfield Sierra Remote Observatories 



Figure 18-1: LCROSS spotted in orbit from 
the ground (2009-06-29)! LCROSS in the 
upper-right quadrant streaking toward the 
two stars in the image. Image courtesy 
P. Mortfield, Sierra Remote Observatories. 



Figure 18-2: LCROSS look back toward 
Earth and the Moon during an “Earth Look” 
calibration of the instruments, 2009. 
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It was no longer enough to remedy 
the original problem - we needed to 
assure this could not happen again. 


I described the LCROSS “Bad Day” in a previous 
chapter. Here I want to focus on the great tension 
we felt at the time on how to proceed to save the 
mission. After the discovery of the dramatic 
propellant loss, the LCROSS ops and engineering 

teams were singularly focused on addressing what went wrong and completing the mission - as they 
should’ve been. The team wanted to work the solutions immediately and get back to the original Concept 
of Operations (ConOps) as soon as possible, as they had trained with those original procedures and were 
well-rehearsed. In a way, those ConOps were the safest way to continue the mission. Unfortunately, the 
reality was that the landscape is simply different following a major on-orbit anomaly. The context of the 
mission with its new, low propellant margins had completely changed, as had the feelings of the 
stakeholders who seemed to have forgotten that this was a Class D mission. LCROSS survived a near- 
death experience, the world was looking and waiting for that big impact event, and those stakeholders who 
wanted to see the LCROSS Class D approach succeed were nervous. The difference between how the 
team was trained to operate and the needs of the post-anomaly stakeholders created great tension. 


Everyone really did have a good understanding that given the little remaining propellant, another 
propellant-related failure would doom the mission. With so little propellant remaining, this changed the 
nature of how we could operate. It was no longer enough to remedy the original problem - we needed to 
assure this could not happen again. The difficult question is to what degree we take extraordinary 
measures to protect against any further propellant emergency. This cannot fail approach to the remaining 
mission stood in direct conflict with how the mission was formulated and selected and how the team was 
trained. It was also an excellent indicator of how much the LCROSS mission context had 
evolved. . .LCROSS went from likely to fail in the minds of some, to hey, it looks like they ’ll actually make 
it, to, this mission is too important to fail. 


...the evolving stakeholder 
conservatism following this 
incident caught both me and 
the LCROSS PSE offguard. 


At this time of heightened risk sensitivity, I found that the 
PM/PSE relationship was more critical than ever. The 
programmatic and technical trades were front-and-center 
and the PM and PSE were running point on them. In the end, 
the PM makes the call on forward action, but a wise PM 
makes sure he fully understands the options, the technical 
risks, and then weighs those against the broader programmatic and political landscape. I have to say the 
evolving stakeholder conservatism following this incident caught both me and the LCROSS PSE off guard. 
While he and I were fonnulating our plans on a prudent and balanced way forward, we were surprised to 
see much more conservative views coming from the stakeholders. Requested new technical activities, 
including analyses, simulations, and flight software belt-and-suspenders functionality were not supported 
by the LCROSS PSE or PM, as they were too taxing on the team who already had their hands full with the 
anomaly. The problem was that our small Class D team was simply not staffed for long-tenn handling of a 
spacecraft with near 24/7 coverage. The burdens of . , , . . . . , . 

AA ... , At what point does solving one 

additional engineering studies and simulations, r ® 

implementation of the new, conservative ConOps problem Only Cause another? 
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protocols, while keeping on top of the mission planning 
for the eventual impact of the moon, were too much to 
carry through the end of the mission. Whenever a mission 
experiences difficulty the risk position of the project needs 
to be considered. At what point does solving one problem 
only cause another? The PM needs to navigate this 
situation very carefully; you cannot simply ignore the 
stakeholder demands, but you must also maintain your 
stewardship and responsiveness to the mission and team. 

Here’s a more detailed look at the propellant anomaly 
context: 

♦ LCROSS had just survived a major loss of 
propellant due to an on-orbit anomaly. The mission 
had been moving along swimmingly until this 
point, but with this incident LCROSS declared a 
“state of emergency” with the NASA’s DSN, giving 
LCROSS ground station priority over other 
missions in the Agency. 

♦ While LCROSS had stopped the excessive 
propellant consumption, and had determined that the 
mission was not lost, it had little if any propellant 
margin to finish the mission. Propellant-hungry, non- 
required science activities would likely be eliminated 
from the forward plan in order for the mission to carry enough propellant to execute final targeting 
thruster firings needed to hit the right spot within the right crater on the moon. 

♦ After some incredible work by the team over the next few days, all while moving to a 24/7 
coverage of the spacecraft (something we did not have the staff to sustain), LCROSS had 
implemented fixes to the root-problem that had caused excess propellant consumption. The team 
had also implemented a belt-and-suspenders propellant usage monitor to assure that the remaining 
propellant didn’t get squandered by an anomaly while out of view of the mission operators. 



Figure 18-3: Preliminary Lunar mosaics 
using the visible camera and a mid-infrared 
imager (MIR1). Images taken approx 8500 - 
9500 km from moon. (Colaprete, 7-1-2009.) 


At one point I witnessed an 
operator's eyes slamming shut and 
his head bobbing while on console. 


♦ Additionally, the project had modified its 
ConOps plan to implement a 24-hour 
coverage policy to make sure we were 
watching the spacecraft at all times; however, 
with this level of coverage, the operations 
team was getting very, very tired. At one point I witnessed an operator’s eyes slamming shut and his 
head bobbing while on console. The LCROSS operations team was never designed for this level of 
sustained support, for cost reasons. The original LCROSS ConOps plan called for two operations 
teams with spacecraft communications passes for 3 hours every three days. This modified ConOps 
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plan had two full shifts of coverage, putting the two flight teams on console for 13 or more hours per 
day, plus additional time spent preparing for future passes and activity planning while not sitting on 
console. This was clearly not sustainable. As the flight director put it in an email to the mission ops 
manager and me: 

My concerns are the following: 

1) Even on our pre-anomaly schedule, the team was very busy all the time preparing for events 
(ASR ’.s' [Automated Sequence Reviews], CAM’s [Command Approval Meetings], coordinating, 
etc.). This DSN [Deep Space Network] contact schedule is four times busier. 

2) We are already at a low, having worked more or less three weeks with only one day of rest 
apiece. 

3) DSN contacts are at all hours of the day. This will fragment the team, greatly diminishing the 
time we can spend coordinating and working as a full team, and exhausting anyone working 
odd hours. 

4) To date, DSN contacts have required the participation of a flight controller and flight director. 
This schedule will take those four [Shift A & B flight controllers and flight directors] out of the 
running for other activities. 

5) The rehearsal is overlapped by 8 DSN contacts, plus 2 immediately before and afterwards. 

We ’ll need to outsource coverage of ALL of those passes to adequately perform the rehearsal. 
Rehearsal preparation will be extremely difficult. 

In my view, following this schedule was introducing a massive risk for team health, morale, and 
readiness for impact, not to mention preparation for ongoing events like TCM 6 [Trajectoiy 
Control Maneuver 6], Earth Stare, Cold Side Bakeout, etc. ” 



Figure 18-4: Rendering and actual image 
of the LCROSS Centaur separation from 
the shepherding spacecraft, in advance of 
lunar impact, 2001-10-09. 
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He was right. Not only was this an incredibly taxing time for the team to babysit the spacecraft, but forward 
work on the lunar impact event of the mission was not getting done. The LCROSS PSE and mission 
operations manager (MOM) felt that we had a serious risk of a human operations error, given the weakened 
condition of the operations team. I agreed and was even starting to think that the likelihood of a human error 
was greater than the likelihood of another spacecraft issue resurfacing. Further, from a political perspective, 
it would likely be a more costly problem for the team and NASA-ARC if the LCROSS mission were to fail 
from a preventable human error than from a spacecraft malfunction or issue. 

LCROSS was never designed to eliminate risk, as that 
was unaffordable; but the local stakeholders were 
pushing us in that direction with the spacecraft while 
growing a higher-likelihood risk of the operations 
team making a mistake. If you squeeze one side of the 
risk equation too hard, risk appears somewhere else 
The stakeholder chain felt that losing a mission (Class D or not) was so unforgivable, that we should fly 
the whole mission into the moon under the declared spacecraft emergency and 24-hour team coverage. 
The PSE, the project team, and I agreed the stakeholders plan was not the way to go, leaving us at odds 
with the stakeholders - and with me as PM, in a dilemma. 




...one of the LCROSS team 
members asked, “ Can we just 
ignore what the Center is telling 
us to do?” That is how bad it got. 


Figure 18-5: LCROSS shepherding spacecraft MIR camera image of the 
Centaur impact plume rising from the Centaur crater, 2009-10-09. 
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The PSE and the PM have escalation paths through the line and programmatic chains of command as 
needed. The PSE can go through the Center Chief Engineer and/or the Agency Chief Engineer with his 
concerns, and I could go to the program manager to elevate my concerns. Ultimately, issues would be 
disposed by the Associate Administrator, but this comes with political consequences. Meanwhile, the 
team was exhausted and questioning what some of the stakeholders were requesting. I saw this during a 
team planning meeting wherein I explained the Center’s position on what they required to move forward; 
one of the LCROSS team members asked, “Can we just ignore what the Center is telling us to do? ’’ That 
is how bad it got. 


This was one of the most challenging times The t‘\ |KTit‘IH’t' of flying H C lUSS D mission 
for the lcross team and for me. The taught the LCROSS team that it must 

project Technical Authority (ta) (the mil ke decisions based on risk, not fear. 

LCROSS PSE) was in disagreement with 

the Center’s Technical Authority (Center Chief Engineer). Because the LCROSS PSE was sitting on 
console when he wasn’t sleeping off-shift, he and the Center chief engineer were not always able to 
communicate. This was putting me in the go-between role and I needed to fix that. I needed the LCROSS 
PSE to work his technical reporting chain while I focused on advocacy within my programmatic chain, to 
make the right thing happen. I shared with the PSE, “I’m thinking it is not effective for me to be the go- 
between with the TA on the project (you) and the Center TA. There is a break between what the project’s TA 
sees and what the Center sees ” and I needed them to resolve it. The PSE and I were absolutely on the same 
page with our view of the forward plan. We advocated for the same forward plan in the frequent meetings 
we each had with Center management. The LCROSS PSE replied, “As you, I'm a bit frustrated that 
management has become so overly conservative. It has increased our risk in other areas and has not set a 
good precedent for flying future missions. 

There are many spacecraft flying right now 
with zero fault tolerance, including LRO, 
and you don 't see them holding onto a SC 
[spacecraft] emergency. ” This comment 
points to the fact that the spacecraft 
operation was returned to its before- 
anomaly condition, and even better with 
the various monitors we had put in place. 

The LCROSS PSE and I had many a 
philosophical discussion during these times 
and concluded that at that point, some in 
the Agency did not have the belly to 
support Class D missions. If nothing else, 

the experience of flying a Class D mission Fi 9 ure 18 ‘ 6: Thermal ima £ e ° f J- CR0 ? S <? entaur 

. impact as seen the LCROSS shepherding 

taught the LCROSS team that it must make spacecraft mid-infrared camera (MIR1 ). Hot 

decisions based on risk, not fear. impact site is the blue dot near the arrow tip. 
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I became increasing frustrated with what I felt was a strategic mistake in the Center’s request that we stay 
in “spacecraft emergency” to be sure we got no propellant-based mission-ending surprises. The dilemma: 

♦ The LCROSS project team was essentially asked to eliminate spacecraft propellant risk at all costs. 
This resulted in growing risks associated with operator error. 

♦ Operator or planning errors on a mission are preventable and if LCROSS were to experience one, 
that would do more harm to the Center’s reputation than another spacecraft technical issue; it would 
be seen as a freshman mistake by an inexperienced team. 

♦ Staying on spacecraft emergency „ „ . . . 

harms other missions who are Of COUFSC yOU WCFC SUCCCSSflll... yOU had 

continuing to surrender their own to have the Agency hold your hand all the 

mission time for the lcross wa y in”. . . . This was unacceptable. 

“emergency.” At first the community 

was very understanding, but as we went into the second week with no sign of coming out of the 
declared emergency, the community started getting restless. In effect, LCROSS had inserted itself 
into the ConOps of many other missions due to our needs for their communications downlink time. 
My concern was that if we flew the remainder of the mission under declared emergency, this would 
harm the perception of the project, as well as set a very bad precedent for future mission behavior. 

♦ Even with a completely successful mission, and the discovery of a veritable frozen Lake Erie of 
water, the community would say, “Of course you were successful. . .you had to have the Agency 
hold your hand all the way in”.... This was unacceptable. 

We needed to get off spacecraft emergency soon. We had the controls in place and the team needed to 
reduce their hours and get back into a sane sleep cycle so that we could keep the operator error risk in 
check. This is no different than how we ran risk management throughout the entire project: attempting to 
establish a prudent balance. We did not eliminate risk; we managed it. 

I was able to convince the Center TA that the team had to get some sleep, which meant we needed a way to 
monitor for a spacecraft issue without requiring the attention of the ops crew. As noted previously, with the 
excellent help of the DSN, we arranged for the scheduled ground assets to continue to monitor LCROSS 
during communications passes, but we utilized a failsafe to be an indicator if a spacecraft problem 
surfaced. Only then would we wake the LCROSS operations team to react to a discovery. This struck a 
middle ground in that the operations team could avoid frequent uplinks with the spacecraft, relying on the 
fault management system to indicate a problem. Only very simple monitoring by the ground stations and 
DSN was required. Additionally, we came up with a secure means for monitoring spacecraft telemetry 
from remote laptops. This was a substantial time saver because our ops team could monitor spacecraft 
telemetry at the beginning of a pass, just to see if everything was fine, while at home. This saved the 
operations team time that would have otherwise been required to commute into work, breaking the sleep 
cycle. Monitoring at home enabled a quick check when required, and then resuming sleep. 
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Figure 18-7: Crater floor before LCROSS 
Centaur Impact. May, 2009 via Goldstone 
Radar of Cabeus Crater, courtesy of Barbara 
Wilson, JPL. 



Figure 18-8: Crater floor after LCROSS 
Centaur Impact. Nov, 2009 via Goldstone 
Radar of Cabeus Crater, courtesy of Barbara 
Wilson, JPL. 


Meanwhile, I continued to advocate for the dropping of spacecraft emergency with my management chain. 
I felt we had sufficiently addressed the root problem of the propellant anomaly, and had a propellant 
monitor in place looking for excessive use and turning off the thrusters if a threshold was reached while 
out of view. I felt there was no longer justification for being on spacecraft emergency. On about my fourth 
try with Center management I was told, “If I were you, I would fly LCROSS all the way into the moon with 
continuous [spacecraft emergency] coverage, using whatever help you need to accomplish that. If you lose 
the mission while you are not looking, you will forever kick yourself.... However, I long ago learned not to 
override the commander in the field 
as he best knows the conditions on 
the ground, the morale and health 
of the troops, and so he is in the 
best place to judge the way 
forward. So it is your decision. ” 


This was a leadership moment for 
the NASA-ARC Center Director - 
a bold and trusting response 
empowering me and the team. With 
this, I had the LCROSS team adopt 
a modified ConOps approach that 
released our declaration of 
spacecraft emergency. We no 
longer needed continuous 
coverage, making use of DSN for 


LCROSS 

Crater 

(Centaur) 
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Figure 18-9: NIR Camera image of LCROSS Centaur 
Impact crater @ 14 km above surface, 2009-10-09. 
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quick heartbeat monitoring passes of LCROSS every nine hours or so, with the team downloading actual 
health telemetry every 24 hours. This was the break the team needed, as they could resume planning for 
the LCROSS impact and get some sleep. 

I later confirmed that there were indeed some Top of the ninth. ..HO OUtS... l-run lend, 
bitter feelings forming by other missions (jg LC ROSS! 

regarding the LCROSS declaration of q 

spacecraft emergency, confirming my original 

concerns. That frustration was understandable, given the nature of how spacecraft emergency works, and 
it was a good thing to drop from that emergency declaration and resume our new normal ConOps plan. 
This plan still had heightened spacecraft monitoring, but achieved it by negotiating with other missions 
for short, available periods of time and not through a spacecraft emergency hammer. 


Figure 18-10: LCROSS 
Lunar impact plume 
dimensionally overlaid on 
satellite image of the San 
Francisco Bay Area. The 
plume is the irregular cloud 
shape near the center of the 
figure. Image courtesy of T. 
Colaprete, NASA-ARC. 



148 



Chapter 18 — Managing the Middle 


As we headed into the final week before impact, having navigated our propellant reserves problem and 
carefully managed stakeholder needs all the way in, I sent an email to the team which read: 


From: Andrews, Daniel R. (ARC-PX) 

Sent: October 2, 2009 
To: (LCROSS team) 

Subject: You feelin' it yet? 

N240 (Our Ops Center building) is getting an exterior clean-up; I'm getting more coordination/logistics 
phone calls than any time since launch; and the family is looking at the moonlit sky a little differently.... 
Oh yeah, and we rehearsed hitting the moon yesterday. ..hitting it twice... 

Top of the ninth. ..no outs. ..1-run lead. 

GO LCROSS! 

-D 



Figure 18-11: Space.com “Top 10 moon crashes” (prior to LCROSS launch) with LCROSS ranking #10. 


149 




Chapter 18 — Managing the Middle 


150 



Chapter 19 — Managing the Big Decisions 


A project manager sometimes feels like a firefighter in that he is having to make many quick decisions 
every day at sometimes an instinctual level since he doesn’t always have all the information needed. 
There are a few times, though, when you have to make a very well-informed “big” decision which could 
strongly shape the future of the project and mission. These big decisions will have supporters and 
detractors no matter which way you go, so you will feel the pressure. . .and your decision may be wrong. 


I want to share what I consider to be the most 
difficult and important judgment call I had to 
make on the LCROSS project. I think it is 
important because it went against the grain to 
a number of people, but to this day I believe it 
was the right decision. LCROSS actively 


We always worked to make the best 
decision with what we knew at the time. 
There is no shame in that. The only 
shame is in being unwilling to revisit 
your decisions when situations change. 


managed its risk position all the way to the 

end. There are few things I would do differently, because we always worked to make the best decision 
with what we knew at the time. There is no shame in that. The only shame is in being unwilling to revisit 
your decisions when situations change. 


The LCROSS team had the opportunity to experience this firsthand while going through spacecraft 
testing in I&T. By the time your spacecraft is in I&T, you should no longer be changing things; this is 
where you learn the nature of the system you built. That’s why it should come as a great surprise to hear 
that this risk-tolerant mission decided to go back into flight boxes that were already tested, verified and 
integrated onto the spacecraft to do some more testing! This was blasphemy. 


Some background: The project was carrying a bottom-right 
corner risk through the latter half of its existence. By 
bottom-right corner, I mean a risk in the project 5x5 matrix 
(explained earlier) which has a very low likelihood of 
occurring, but a very high consequence of fatally-harming 
the mission. It’s one of the tougher types of risks to handle. 


TJ 

O 

O 



In this scenario, the root of the issue was a capacitor... a 
particular capacitor from a particular vendor which had 
experienced some failures. The suspect capacitors were 
found to be traceable to manufacturing flaws and could be 
found in various part lot codes. The project had already 
confirmed that we did not have any capacitors from suspect 
lot codes on our spacecraft, but we did have different lot 
codes of precisely the same part and manufacturer in 
hundreds of locations on the LCROSS spacecraft. They could be found throughout the avionics system. 
In order to understand our risk exposure, we undertook a study of the circuits in which these capacitors 
were located and found that the vast majority of these capacitors were in benign places in the circuit; 
locations where they were used to simply reduce electrical noise and a failure of the capacitor would not 


i 


3 4 5 

Consequences 

Figure 19.1 : 5x5 Risk Matrix with 

bottom-right corner risk illustrated. 
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critically impair the circuit. There were a couple of 
circuits, however, where the loss of the capacitor 
would fail the circuit, damaging some other 
component and critically harming the mission; one 
of the locations could cripple the spacecraft’s 
ability to communicate. 


It should come as a great surprise to 
hear that this risk-tolerant mission 
decided to go back into flight boxes 
that were already tested, verified 
and integrated onto the spacecraft! 


The systems engineering team did a considerable amount of investigation with vendors whose sensitive 
components were in the same circuits with the suspect capacitors to understand the risk. Of course on the 
most sensitive items, the vendors were not of much help because we were in effect asking them, “//ow 
long can we use your parts in an out-of-spec capacity before they will die?” You won’t receive good 
answers from vendors on what will happen when you abuse their parts.... The lingering question was, 
while our circuits were all working in box-level testing and systems level testing with no indication 
anything was wrong, could it be that one of those critical capacitors was already failed, and the sensitive 
components are already under stress, potentially failing at any time? 


Our decision at the time was to accept the residual risk with the capacitors. We did not have capacitors 
from the lot code in question, the instances where that capacitor would cause a critical fault were few, 
and the system had been built and tested satisfactorily. We assessed the risk as a “VL” (very low) 
likelihood of having a “VH” (very high) consequence (i.e., mission-ender). Welcome to the bottom-right 
corner. As a Class D mission with cost and schedule constraints, we could reasonably choose to accept 
the residual risk that one of our critical capacitors would fail. In fact, tearing down the avionics, locating 
the suspect capacitor, removing and replacing it, reinstalling the affected circuit board, and re-dressing 
the electronics boxes carried risks as well. . .not bottom-right comer risks, but probably a couple of green 
risks. At the LCROSS risk management board (RMB) the team decided to accept the risk of having one 
of these faulty capacitors on the spacecraft. It was the right thing to do, given our mission classification 
and cost/schedule constraints. 


This assessment caused some consternation within the stakeholder community, but the majority felt this 
was the right judgment call. It seemed almost heretical to go back into a spacecraft electronics box that 
had already passed its environmental testing, thereby undoing all the testing heritage. For the next several 
months and several RMBs, this risk remained a bottom-right comer risk; we all knew what it was and 
were fairly comfortable with it, having reasoned our way through the logic. During that time, worked 
progressed and we were fully involved in I&T, and our launch date moved out. The manifest for the 
launches out of Cape Canaveral changed and LRO/LCROSS were pushed to the right. In fact, this would 
happen several times over the subsequent weeks and months changing our needs dates. One morning I 
found myself reflecting on our new project context and realized it was very different than it was earlier 
and some of the decisions we made may no longer make sense. I went down the hall to chat with my PSE 
about this thinking and as it turned-out, he was at the same place with his thinking. Here’s an email I sent 
to the stakeholders: 
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“Inside NASA*s Plan to Bomb the 
Moon and Find Water 



Short on time and tight on money, 
a team of NASA engineers aims to solve the 
mystery of lunar ice - by crashing its low- 
budget kamikaze spacecraft into a crater" 


Figure 19-2: Popular Mechanics article on LCROSS showing the LCROSS 
spacecraft getting ready for Thermal Vacuum Chamber (TVAC) testing at 
Northrop-Grumman in CA. LCROSS PM Daniel Andrews shown on the right. 


From: Andrews, Daniel R. (ARC-PX) 

Sent: Tuesday, October 28, 2008 7:26 PM 
To: [program office stakeholders] 

Cc: [NASA-HQ stakeholders] 

Subject: LCROSS strategic risk decision 
Importance: High 

All- 

One of the responsibilities I have in my role is to balance the various trades of cost, schedule, 
requirements, and then risk as an overlay across all. The decisions we’ve made on this project have been 
sound, and I stand by them, as they were proper thinking within the context of that moment. 

So now that we’re looking toward launch, it’s a good time to do a gut-check on where we sit with our 
remaining risks. 
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This leads me to the [vendor] capacitor topic, which is the project’s #1 risk. Our Risk Management 
process characterizes this risk as a VH (Very High) consequence (mission ender) with a VL (Very Low) 
likelihood ( <1% chance, although this likelihood is somewhat hard to quantify). To date we have been 
pushing to get this risk “accepted,” as there is little we can do to further reduce the likelihood (can't drive it 
below VL); however, if on mission we were to lose COM with the LCROSS spacecraft, one of the FRB 
[Failure Review Board] topics would be to assess what went wrong. The [capacitor] Risk will be noted as 
a VH risk that was accepted. The notion of accepting it will be poked at a bit, and inevitably it will come 
down to, “Did you do everything reasonable (not everything possible) to mitigate this risk?” The course 
we've taken to date has been sound & wise, as we related the risk to its context at the time. Back at 
FlatSat, C&DH was on the critical path to a 10/28/08 launch and all our risk was in front of us. During the 
time of the MAC card repair the spacecraft was on critical path, with a Dec-08 launch (likely), and still 
some risk in front of us. 

So where are we now? We have now have a completed spacecraft, off the critical path, the bulk of the 
spacecraft risk has been retired, we are launching no earlier than 4/24/09, and we have positive 
reserves.... This is a very different context than in the past. We have a VH risk that we could do 
something about. ..and it is hard for me to justify doing nothing about our#1 project risk. 

So I've spoken to my SE, several technical folks at NG, and just this morning called a meeting with ACD 
Worden, ACE Panontin, IRBC Klupar, and Code P line management to put forward the idea of lowering 
the R5 panel and confirming the operation of the subject capacitor feeding the SCOMM FPGA power, and 
the cap feeding the HKIO FPGA power. Pete Worden's take was that this is the prudent thing to do, given 
the importance of this mission to the Agency, and the availability of time and money to do so. 

With this affirmation, I will be having NGST perform this test on the quality of the power going into the 
subject FPGAs, thereby confirming the health of the capacitors in question. We expect that the caps/ 
voltage will show everything is fine, at which point we will follow the already proven procedures for 
buttoning everything back up and close the risk. If we do discover that there is a latent cap failure, then 
we will need to make remediations. The exact steps for this remediation are already being discussed. I 
am also establishing a cost threat to support the cost of this effort. 

As a result of this testing, we may spawn a new, lower-level risk associated with the activities on the R5 
panel and the C&DH chassis. We believe we have already demonstrated the viability and means for doing 
precisely the type of work, as demonstrated with the previous MAC card work on the same panel. 

In the end, this decision enables us to close the project #1 risk one way or the other; we either confirm that 
there is no risk as the cap is functioning properly and there is no elevated voltage stress to the FPGA, or we 
discover the cap has failed and we remediate it to goodness, thereby closing the risk. As I stated in a 
previous email when LCROSS received formal notification of the launch date change, LCROSS is 
navigating to its current context. The fact that today (10/28/08) was the original LRO-LCROSS launch date 
is timely, but not of consequence, as we are planning our approaches and risk opportunities based on our 
new launch date directive of 4/24/09. 

-D 
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Our Class D designation 
enabled us to take risks, 
but if we had the means to 
do something about the #1 
project risk, and didn't, 
that would be a shame. 


The story is self-explanatory. The risk didn’t change, but the 
context in which it existed did. The thought of the Failure 
Review Board looking back at a LCROSS failure was difficult. 
Our Class D designation enabled us to take risks, but if we had 
the means to do something about the #1 project risk, and didn ’t, 
that would be a shame. 


So, we opened up the box with great care and our NG C&DH 
electronics engineer very carefully extracted the relevant 
boards and hand-carried them over to an awaiting electronics bench, where he meticulously connected 
the relevant scoping points, and powered up the card. It was a moment of great tension, as we were 
committing what many considered heresy, but I knew it was the right risk judgment call. Once everything 
was powered up and the scope traces monitored, the answer was... the capacitors were fine, performing 
just as they were supposed to. 


Time to celebrate? It was an interesting feeling. It was good to kn ow the condition of the circuits and we 
had now officially retired that bottom-right corner risk, but it wasn’t really a moment to celebrate. It 
simply yielded the peace of mind of turning an u nkn own into a known. Some people would later second- 
guess the decision, stating, “See, you didn’t need to go back in there,” and of course if we had found 
something, others would have said, “ told you so.” In either case, we now needed to carefully reinstall 
everything, button up the boxes and accept the residual risk associated with these testing activities. 

I think this capacitor risk example is one of the shining moments of the LCROSS project. The details 
make the story interesting, but it really comes down to the process - the approach we took. It is an 
excellent example of continuous risk management. We had carefully dispositioned a known risk for a 
number of months. Accepting that risk was the right decision at that point in the programmatic flow. 
When something within that programmatic context changed, we had to change our thinking - 
unfortunately right as almost everyone had gotten comfortable with accepting the risk, we changed 
course. Not because of hesitancy or pressure, but because it was now the right thing to do.... Be aware 
and be willing to change course if that is the right thing to do. 


When I reflect back on this time, I wasn’t “shaking in my boots,” despite knowing how very important 
this decision was. It was one of those “at-peace” moments where regardless of the condition of the 
capacitor, it was the right thing to do. If you establish a culture of true risk understanding in risk 
management it can bring clarity. Our LCROSS experience revealed that proper risk management can 
actually help you understand not only what you should do, but what you should maybe not do. 
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It’s 3:30 am PDT on Saturday, August 22, 2009, and my cell phone 
just rang at home. The LCROSS team was about to begin a third 
cold-side bake maneuver, which I was going to attend later that 
morning. The call is from the LCROSS mission operations manager 
(MOM), who tells me that they just got acquisition of signal (AOS) 
with the LCROSS spacecraft and discovered that a huge amount of 
propellant had been consumed while we were out of view of the 
ground stations - some 140 kg was lost, leaving us with a mere 60 
kg to finish the mission. At the moment, the thrusters were still firing very frequently. It was not clear if 
we had enough propellant to finish the mission, but it was clear that if we hadn’t scheduled this cold-side 
bake activity when we did, we would have consumed all the propellant and lost this 
mission. . . LCROSS ’s Bad Day. 

I got dressed and headed into the mission operations control 
room (MOCR), where I learned the spacecraft had an 
inertial reference unit (IRU) fault. The spacecraft control 
system used the IRU to track the relative orientation 
velocities of the spacecraft, which enabled us to control the 
its position very well. Since the IRU had faulted, the 
autonomy and fault management (A&FM) system had 
kicked in and switched to using the star tracker (STA) for 
the rate (velocity) feedback instead of the IRU. For some 
reason, this change drove the attitude control system (ACS) 
to fire the spacecraft thrusters an extraordinary number of 
times, consuming a very large amount of propellant. The 
operations team executed a power-cycle on the IRU and 
that stopped the excessive thruster firing, returning the 
spacecraft to normal operation. Figure 20-1 : The Lonely Moon. 

We had stopped the bleeding of continuous thruster firings. Now we had to understand exactly what had 
happened (root cause) and how to prevent it from happening again. Some of the greatest stakeholder 
management challenges would occur during this time. To convey the nature of this event, I have provided 
excerpts from the LCROSS Flight Director’s blog (August 21/22, 2009): 1 

“ Shift A came into the Mission Operations Control Room (MOCR) on Saturday with a big agenda 
on LCROSS - to execute a third 'Cold Side Bakeout' to rid the Centaur of more ice trapped in 
its skin, and to perform an Omni Pitch maneuver to flip the spacecraft to re-point the primary 
omni-directional antenna at the Earth, all under a tight schedule with little margin for error. 

The shift started at 2:25 AM Pacific time and we acquired spacecraft telemetry at 3:25 AM. 

In the MOCR, each Flight Team operator sits behind a set of computer monitors that display 
fields of telemetry data - numbers, status indicators, etc. that indicate the health of 
LCROSS. Many of the telemetry data fields show up in stoplight colors - green, yellow, or red, 
to indicate whether the value is "nominal" (green), or approaching an emergency state (yellow). 



If we hadn't scheduled 
this cold-side bake 
activ ity w hen we did, we 
would have consumed 
all the propellant and 
lost this mission... 
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or is in an emergency state (red). Typically, telemetry comes up entirely green. That day, 

LCROSS came up with lots of yellow and red alarms. We had never seen anything like this in 

flight. The summary of our observations: 

• The Inertial Reference Unit (IRU), our onboard gyro, and primary means of measuring 
rotation rates around each axis for attitude control, had faulted; 

• The Star Tracker (STA) had replaced the IRU in providing rotation rate estimates to the 
attitude controller; 

• Spacecraft rotation rates were periodically exceeding yellow high alarm limits (rotating too 
fast); 

• LCROSS was firing thrusters almost continuously; 

• The propellant tank pressure sensor (our primary means of determining how much fuel we 
have left), indicated we had consumed a LOT of propellant. 

In short, we were in serious trouble, and needed to make corrections immediately. ” 


This was certainly a bad day, but I was tremendously impressed by the flight team’s perfonnance. They 
were calm and executed the right troubleshooting procedures. They were autonomous and efficient, 
despite my being there in the room. I knew my role was to watch for the Level-3 requirements threats, 
watch for team roadblocks and breakdowns, and be the voice to the stakeholders. Returning to the blog: 


“The anomaly had caused LCROSS to automatically switch from using the IRU to the Star 
Tracker (STA) for measuring spacecraft rotation rates. However, when we studied the output 
of the IRU, it was producing good rotation rate data (or just "rate data" for short). So, we 
ran through the IRU recovery procedure to re-engage the IRU with attitude control, and sure 
enough, that finally returned the thruster firing profile to normal and bought ourselves time to 
think. 

In summary, a spurious, short-lived error on the IRU was interpreted as a more serious fault 
by the spacecraft fault management system, resulting in a switch to a backup rate sensor 
(STA). That rate signal was noisy causing over-control by the attitude control system, 
resulting in a great deal of propellant consumption. LCROSS detected the associated tank 
pressure drop, but with no option for disabling thruster control (the only means of keeping its 
solar array pointed to the sun), and no ability to determine the specific nature of the pressure 
loss, the spacecraft fault management system performed steps to stop a leaking thruster 
(another potential reason for a pressure drop), and to power-up its transmitter to "phone 
home" to warn the operations team that there was a problem. However, this call could not be 
detected over the Southern hemisphere, since there were no DSN assets that could "see" 
LCROSS in that location, so the call was missed. ” 

The LCROSS project declared a “spacecraft emergency” with NASA’s Deep Space Network (DSN). With 
this declaration, all missions using the DSN have a professional agreement to yield their ground asset time 
to the ailing mission. LCROSS was then able to get continuous communications with the ground, limited 
only by geometric constraints of the spacecraft to Earth (as noted in the blog entry above). 
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As it turns out, one of those geometric outages was again coming over the southern hemisphere, so we 
needed to put some protections in place, just 10 hours after discovering the anomaly. Our plan was to 
update the persistency with which the IRU fault was monitored, to be sure a spurious fault would not trip 
us into another costly propellant consumption situation. . .then we went dark again. 

When we came out of the communications outage, we discovered there had been no further incident - we 
had made it through, but this was the beginning of a new operational environment for LCROSS. Clearly 
a sequence of activities would be required to move from an anomaly to recovery - not unlike triaging the 
injured on the battlefield. Here’s how I saw the steps: 

♦ Stop the bleeding. The mission is over if you cannot stop the elevated rate of consumption of a 
limited resource like propellant; we were effectively bleeding out. Electrical power can be renewed 
through the solar arrays via the sun, but propellant cannot be refueled on-orbit. We needed to stop 
the propellant consumption ASAP, even if we did not immediately understand the root cause. We 
were able to do this in our immediate triage of the incident by resetting the IRU upon acquiring 
signal. The next step was to make it through the night. 

♦ Make it through the night. We 

stopped the bleeding but we needed to 
survive upcoming, known 
communications outages due to 
orbital geometries. These unavoidable 
outages were a concern to the 
operations team, who had to make it 
through without another incident. We 
needed a way for the spacecraft to 
detect when excessive firing was 
happening and then somehow prevent 
further consumption from occurring 
without human intervention. This 
didn’t necessarily address the root 
cause of the problem, but we hoped it 
would limit our risk exposure through 
the night. 

♦ Assure the long-term health. At this point, you’ve stopped the bleeding and figured out some 
means of getting through periods of imminent danger. Now, how do you make sure you can finish 
the mission? What are the tasks remaining and the risk exposure in executing them? We had to look 
hard at how far we could go with analysis, simulations, and other risk mitigations. It was easy to 
want to assure this could never happen again, applying belt and suspender fixes, but how does that 
work compare with the workload risk to the operations team? The operations team was becoming 
exhausted from the long hours continuously monitoring the spacecraft and executing mitigations, as 
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Figure 20-2: Following the propellant anomaly, the 
attitude control system (ACS) algorithms were 
adjusted to be miserly with the remaining propellant. 
I had calculated that we got LCROSS’s usage down to 
-33 thimbles of fuel a day (40grams/day of hydrazine) 
or the equivalent of a tenth of a can of soda: 
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well as simulations and forward planning. At what 
point does the human-error risk become greater than 
the technical risk associated with the spacecraft? 

♦ Address the root cause (if you can). Discover the 
specific cause or reason for the incident in the first 
place. Can anything be done to prevent this in the future? Can we fix it, or only avoid the 
circumstances that led to it? Can we fence off the problem and stay outside the fence operationally, 
even if we don’t fully understand what is happening inside the fence? 

LCROSS walked through each of these phases. It was clear when we were exiting one phase and moving 
into the next. That is not to say it was easy, and the LCROSS team certainly earned their stripes. In 
retrospect, some people anecdotally noted that the mission was going along too easily before that morning. 
That ended that Saturday morning; this mission anomaly was going to reveal our true colors... what we 
were really made of. 


At what point does the 
human-error risk become 
greater than the technical risk 
associated with the spacecraft? 



Figure 20-3: NASA LCROSS news briefing for the media. (From left to right, Jonas 
Dino, Daniel Andrews, Anthony Colaprete, and Jennifer Heldmann). 
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In my reflections on this time in the mission, it was very compelling to see the new environment of 
LCROSS, and how with that environment came new responsibilities: 

♦ Responsibility to the stakeholders. Stakeholders get very interested after an anomaly; it’s 
understandable. They want to find ways they can help to assure the mission. The morning of the 
anomaly, I acted along pre-planned procedures to call different stakeholders and infonn them of 
what had happened. Shortly after those notifications went out, the NASA-ARC Center Director and 
most of his directors-for arrived at the MOCR with bags of breakfast food and dri nk s. It was one of 
the more surreal images of the mission: executives as waiters, but it was their way of keeping the 
team fed and alert, and the team appreciated it greatly. 

My approach to managing the stakeholders was to provide very frequent updates to all of them of our 
findings and progress. I did this in person for local stakeholders and via email for the broader Agency 
audience, with daily status briefs in the morning by the MOM. The email updates were nearly hourly in 
the beginning, later dropping to updates at shift changes near the end of our emergency. My deputy PM 
John Marmie and I tag-teamed on covering shifts in the MOCR, and when we completed a shift, we 
would write up a summary and publish it to the stakeholders. A compiled transcript of all of these 
updates provides a historical record of our discoveries and our responses, and can be found in the 
appendix of this book. This transcript represents the reverse-chronological listing of all updates provided 
to the stakeholders over the course of the spacecraft emergency declaration. The status was sent to a pre- 
defined email list, which went as high as the Associate Administrator for ESMD and the NASA Chief 
Engineer; I heard it was even making it to the NASA Administrator. It was very well received and kept 
the stakeholders reasonably infonned and comfortable. 

♦ Responsibility to protect the team from local ^ simplicity of the ( lass l) 

distraction. The LCROSS team was of course framework was deeply 

attempting to get back to a more nominal operating challenged during this time, 
scenario as soon as feasible after the anomaly because 

this is what they had trained for. However, some very strong edicts coming from center 
management directed us to take particular actions and put particular controls in place to protect the 
remainder of the spacecraft’s propellant. This challenged the team at a time when they were 
obviously very stressed; some on the team felt these decisions were based not in sound risk 
management, but in fear. The simplicity of the Class D framework was deeply challenged during 
this time. This mission had grown to be very important to many, not the least of which was our 
team, but what really mattered was reason and balance. In the end, these edicts forced us to 
implement controls that moved us down a very good solution path. The next chapter is dedicated to 
this topic and the difficulties of the PM having to “Manage the Middle.” 

♦ Responsibility to protect the team from global distraction. Once you get past the “stop the 
bleeding” triage phase, questions naturally begin to surface about why the anomaly happened in the 
first place. This query formally occurs after triage is complete, allowing the team to focus their 
immediate attention on making it through the night; however, sometimes the stakeholders cannot 
help themselves. The initiation of the problem for LCROSS was an intermittent IRU fault that 
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caused the system to switch over to the star tracker for rate feedback. . .feedback that was 
sufficiently noisy to cause the ACS to fire the thrusters rapidly; however, why didn ’/ we know this 
would happen? This is a fair and reasonable question to ask (and later answer), but it is a very 
dangerous path to walk down while the team is still trying to save a mission and getting very tired 
from these efforts. I had to push back on this line of questioning whenever I saw it because it served 
no one to have the team that we had to depend on get frustrated or distracted. The program office 
helped reinforce this push-back with others. 


You have to stay in front 
of the media and make 
yourself available to 
answer their questions. 


♦ Responsibility to handle the media. When a spacecraft 
experiences an anomaly, media-handling is very interesting 
and important. In short, you have to stay in front of the media 
and make yourself available to answer their questions. The 
traditional media are manageable; they want to know all the 
details, but if they smell any hint that you are holding back, 
they can make sport of you. The blogosphere is a different matter; their “facts” come from 
u nkn own sources and their conclusions are based on personal assessment and even agenda. I have a 
chapter dedicated to this very topic in “Managing the Lunacy,” which includes blogosphere 
assertions about aliens wanting to protect the Moon from LCROSS and having a hand in what 
happened with the spacecraft - alien sabotage. 


We handled the press by updating the project web page, even before a press release. This 
immediately netted a number of calls from the press, which I fielded directly. I conducted seven 
phone interviews in two days, and their stories hit the web shortly thereafter. I was honest but 
careful to not impugn anybody, including the IRU and STA vendors because the facts were still 
unclear (and later revealed that the problems were not with the hardware). But the press 
immediately wanted a smoking gun. 


♦ Responsibility to be the watchful eye. After the anomaly, the LCROSS team, both the operations 
team and the engineering teams, focused on a way forward and the question was what and how 
much to do. While I am an engineer, as the PM, I intentionally let the engineers work through the 
data and come up with possibilities. Engineers are pre-disposed to solving problems (a good thing), 
but I found a tendency to create complex, multilayered solutions in order to stomp out the risk of 
recurrence. The discussion would start in one place, and would then work its way from one 
incremental fix to another, building upon each until the solution extinguished the problem. 


The drawback was that this resulted in a very complex 
series of fixes and patches, which moved the team 
off of its training and operational context and would 
create modes on the spacecraft that had never been 
tested. Such additional complexity actually grows 
risk that the system will either be untestable, or will 
be so sophisticated that it will be prone to operator 


In the heat of battle there 
needs to be a person who just 
keeps wraps on the relative 
risk complexity of solutions. 
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Figure 20-4: One of many LCROSS press conferences. The PM has responsibilities 
to clearly communicate information about the mission. 


error. In the heat of battle there needs to be a person who just keeps wraps on the relative risk 
complexity of solutions. There were probably three different times where I saw the engineering 
team formulate a software means to work around a problem, that just got too complicated. I would 
ask, “Do we need to go that far, or can we live with just the first corrective measure?” More often 
than not, we agreed we could live with the residual risk because the likelihood of occurrence was 
small. This reduced the risk of the unintended consequence. A fair number of missions have been 
lost by smart people doing well-intended things, and introducing flaws into a mitigation. 

♦ Responsibility to watch operations console staffing. The LCROSS team was small, and one of 

the leverages we made was to have the project systems engineer (PSE) staff the systems 
engineering console station. The PSE would take one shift, and his deputy would staff the other. 
The idea was simple - why not put your most competent systems engineer right in the middle of the 
action, where he is of most value? I later realized that having the PSE on console removes him from 
his normal responsibilities. . .which still need to be taken care of. Yes, we benefitted from having 
our lead SE monitor the spacecraft, but being human, he had to sleep and be off shift; thus he was 
less able to participate in important assessment and planning activities. This removed him both 
from one of the operational shifts, and from me - unable to advise me on his recommendations on 
the overall technical status. I would not organize staff this way again. 
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We needed to balance attacking 
the existing problems against 
the growing operational risks 
associated with fatigue. 


♦ Responsibility to watch crew fatigue. Hard- 
working, dedicated people get tired. The staffing 
plan in which LCROSS was designed to operate 
was not the post-anomaly staffing plan. A small 
number of people were covering an extraordinary 
number of hours and while impressive, it was the 
wrong thing to do. Fatigue started to set in and we needed to balance attacking the existing problems 
against the growing operational risks associated with fatigue. This period of time was undoubtedly the 
hardest the MOS team had to endure over their entire time on the project. I saw heads bobbing while 
on console as they fought back sleep; I saw people struggle to complete thoughts during shift 
handovers. There was also growing stress at home for many people who were not only working crazy 
hours, but were zombies the few hours they were home. This only added to the personal stress the 
team members were experiencing. With staff constraints like this, it is important to get the spacecraft 
into a safe position as soon as possible in order to keep from growing operational risk elsewhere. 



Figure 20-5: LCROSS mission operations control room. Flight Director Rusty 
Hunt and Flight Controller Jim Strong at their stations at NASA-ARC. 
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Crew fatigue is an incremental problem, and within a week of the anomaly, I saw signs worthy of 
concern. The stakeholders were apprehensive about the issue at hand: spacecraft fault, and eliminating 
the forward risk associated with it. This was the team’s focus as well, but as with LCROSS risk 
management throughout the project, I had to be mindful of the entire risk space. To combat problems like 
these, I worked with the center stakeholders to attempt to strike a balance. 

The team came up with some clever solutions to improve the fatiguing workload. The first was to drop 
from continuous DSN coverage to having a communications pass within every nine hour period. In 
effect, it was taking how we handled geometric constraint outages and adopting that into the LCROSS 
concept of operations (ConOps). The logic was that since we had put propellant usage limit monitors in 
place in the spacecraft software, we could limit the degree to which we had to monitor the spacecraft. 
The propellant-use monitors worked by stopping further thruster firings (regardless of causing event) 
after a prescribed number of firings. This would put the spacecraft into a free-drift state to preserve the 
propellant, and if the spacecraft drifted too far, the solar arrays would move far enough off-sun for the 
batteries to start to drain. Analysis showed that upon commencement of free-drift, we had approximately 
nine hours before the spacecraft power systems would be in trouble. This was somewhat uncertain, 
however, because the estimate is a strong function of the initial conditions at the point when the 
spacecraft enters free-drift, (at what rate and direction is it drifting?). Nevertheless, something had to be 
done to address the crew fatigue. 

This every-nine-hour constraint did help the team; we no longer needed continuous coverage, but 
periodic check-in was required on the spacecraft three and sometimes four times a day. So the team 
continued to get up at strange hours to come in and check the spacecraft. Sleep patterns were messed up 
and the team still had forward planning work to complete for the next maneuvers. I advocated the 
following further modification to the ConOps approach in an email to the stakeholders on 9/7/09: 


From: Andrews, Daniel R. (ARC-PX) 

Sent: Thursday, September 7, 2009 
To: [stakeholders] 

“Here is my understanding of what we discussed in the phone call earlier today, and look for your 
concurrence with this approach in advance of advocacy to [Center Management]: 

* LCROSS staff is getting wiped-out by the new ConOps approach requiring spacecraft AOS 
[Acquisition Of Signal] every 9-hours or less as it is amounting to 3-4 AOS a day. Under this modified 
ConOps approach, spacecraft risk has gone-down, but crew-fatigue and product-development risk 
(for finishing the mission) have gone up. 

• LCROSS is advocating for a further modification of its ConOps to address these increased risks to 
mission success while maintaining the < 9-hour checking-in on the spacecraft. 
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LCROSS proposes modifying ConOps to keep the transponder on at all times whether in view or not. 
When LCROSS approaches LOS [Loss Of Signal] it will set the symbol rate to 1 6kbps (or equiv) and 
leave it in that state as LOS occurs. There is every expectation that when a later AOS occurs, the 
symbol rate will remain at 16kbps because nothing would have changed since LOS. If, however, 
something does go wrong with LCROSS while out of view, safe mode and the Prop monitor will kick 
the symbol rate down to 2kbps, per the existing ConOps. In doing this, any asset that is viewing 
LCROSS will see the symbol rate change and know that something is off-nominal with LCROSS. 

In order to help relieve the LCROSS team from the fatigue of staffing continual LOS/AOS on this < 9- 
hour cycle, we use DSN [Deep Space Network] assets to simply get lock on LCROSS and check its 
symbol rate. If it remains 1 6kbps (where it should be), the spacecraft is fine and the LCROSS team is 
informed of this fact on a non-emergency basis; if however the rate is 2kbps, a call tree is initiated to 
wake/interrupt/mobilize the LCROSS Ops team as there is something anomalous going on. 

The existing ConOps plan with the mitigations already flying on the spacecraft already assure we can 
handle being out of view 9-hours without undue risk to prop.” 


Center management agreed with the approach, provided that during day shift hours, when many on the 
team would ideally be working anyhow, an AOS could actually be initiated by the Ops team and 
telemeter data down to confirm that the spacecraft was fine. We expected to do this anyhow since we 
needed to conduct normal operational activities with the spacecraft (such as VR [virtual recorder] 
playback). This new ConOps approach allowed the rest of the team to maintain a relatively nominal sleep 
schedule and opened up sizeable blocks of time to work on products required to finish the mission. 
Trajectory correction maneuvers, payload calculations and any other required mission activities would of 
course be scheduled in this flow per the baseline plan. 

It was time to try it out, and the first few passes were a little nerve-wracking, but it then became apparent 
this was a great solution, and with the help of the folks at DSN, it allowed this small team to get badly 
needed rest, knowing the spacecraft would let us kn ow if something harmful had happened. Below is my 
status update with the stakeholders, including a further risk reducer; 


From: Andrews, Daniel R. (ARC-PX) 

Sent: Thursday, September 10, 2009 9:16 AM 
To: [stakeholders] 

Subject: Good risk reduction for LCROSS Ops 
All- 

We performed some testing last night during a couple planned LCROSS passes which verify our latest 
planned improvement to LCROSS ConOps by making use of DSN monitoring. Additionally, we have run 
a test of a new remote-monitoring capability improving our effectiveness should we discover a future 
problem with the spacecraft. 
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Recall that existing LCROSS ConOps requires we not go for more than 9-hours out-of-view with the 
LCROSS spacecraft. This has introduced a large burden on the Ops team, around the clock, requiring 
the team establish contact with the spacecraft to assure it is OK. Because only the Flight Directors (2) 
and Flight Controllers (2) are able to actually perform AOS/LOS, this burdens the most burdened people 
on the Ops team and threatens nominal mission planning and execution. 

We have, therefore, tested a new mechanism for checking the spacecraft health by leveraging DSN to 
configure for telemetry receipt for designated unstaffed contacts. At AOS, if they cannot lock onto the 
standard out-of-contact 16 kbps, they will try to lock onto a 2 kbps signal. If that is successful, or if they 
cannot lock onto a subcarrier at all (both of which are strongly indicative of a problem on LCROSS), DSN 
will contact members of the operations team, according to a pre-arranged call tree, who will take 
immediate action. 

We have tested this approach twice on Wednesday evening (9/9/09), while watching from the NASA- 
ARC MOOR. As Flight Director A reports, “we tested this concept on two evening passes, one on DSS- 
27, and the other on DSS-45. In both cases, we designated the passes as “unattended,” governed by 
DSN SOE “Echo” [this is the new monitoring mode Sequence Of Events approach LCROSS developed 
with DSN]. Though we monitored the telemetry from the MOOR, both passes were conducted without 
support from the LCROSS team, with full telemetry receipt, and ranging enabled.” 

It is our intention that armed with this capability we will designate some fraction of our future passes as 
unstaffed to reduce team workload, while assuring the Ops team actually monitors spacecraft telemetry 
once a day as a double check that all is well. 

We are also verifying a remote monitoring capability. During one of the passes described above, Flight 
Director B tested remote, secured, laptop-based telemetry monitoring from home. He successfully 
connected to the backup LCROSS FEDS [Front End Data System] and was able to monitor data from 
DSS-27, proving out the concept. This will improve the Ops team's effectiveness should team members 
get calls during sleep hours and quickly want to see what is happening with the spacecraft (not requiring 
being at Ames to see spacecraft telemetry). In fact, this will likely be an asset used by future missions. 

These two approaches are expected to reduce staff time servicing the spacecraft, lowering the threat on 
our ability to execute the nominal plan. LCROSS is continually attempting to balance risk across the 
remainder of the mission and these are two good steps to that end. 

-D 


So we had an approach that seemed to work well in balancing the risks associated with the spacecraft and 
the risks associated with a fatigued team. Now we just needed to make it through to the planned end of 
the mission. . .hitting the moon. 
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Good Hews for LCROSS 

tottrt M 1 l:M i (awt 


NASA internal Memo: LCROSS has rescinded Its Dedaratlon of Spacecraft Emergency 

list nfght we lifted our defloration of Spacecraft Emergency with DSN with concurrence from the 
Center. We hove Implemented a number of mltifotlom to try to assure that we ore In a more robust 
position Ofatnst propellent lost as we finish the remaining JO days of the mission. Our work is of course 
not done as we continue to navigate out risk for the remainder of the mission, but we need to be aware 
of other risks which have grown through this process. . . namely staff fatigue While our new ConOps plan 
requires we check S'C health every 9 hours, we are looking Into ways to make that monitoring as 
efficient as possible by setting up remote access and by seeing If other assets can monitor for our S/C 
'phone home' If the S/C Is In trouble. ' 



Figure 20-6: NASA Watch Coverage of LCROSS anomaly recovery and spacecraft emergency. Content 
taken from leaked PM email to the team (always be sensitive to what you write and where it could go). 


End Notes 

1. Tompkins, P. LCROSS Flight Director’s Blog entry: 10/4/09, http://blogs.nasa.gov/cm/blog/lcrossfdblog/posts/post 
1254701386237.html 
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One of the most interesting, surprising, frustrating, and even amusing aspects of the LCROSS mission 
has been the media and their interest and coverage of this unconventional project. NASA Class D 
missions, by definition, have a moderate probability of being unsuccessful. This designation is only 
assigned to missions that have tough programmatic constraints, are generally inexpensive, and do not 
draw a lot of attention to themselves. This 

was thought to be the case for lcross at [Vloon as a launching point; moon as a 

inception, but as with almost everything fue , depo| . muun „ s „ destination; moon 
about this innovative mission, the fascination m . _ 

with it grew beyond expectations and it ended “ S * manufacturing |>lnnt...l until 

up garnering a whole lot of attention... not all questions needed tO be ailsw Cl Cll. 

of it what you’d expect. 


One notable angle on the LCROSS mission was its programmatic context. The LCROSS project was a 
grand programmatic experiment to see if a different kind of mission construct was even feasible. Another 
angle on the LCROSS mission, and arguably the more public angle, was the water question. A whole 
community of lunar scientists was interested in the water question, as an important part of the broader 
exploration question; water would be a pivotal element in future human exploration of the moon and 
beyond. Moon as a launching point; moon as a fuel depot; moon as a destination; moon as a 

manufacturing plant... Lunar questions needed to be 
Within the project we would answered. While the space industry media were 

joke that we were simply principally interested in the programmatic angle, the 

hi tiding Otl the tttOOti .with vigor. general media were more interested in this unusual idea 

of water on the moon - how could this possibly be? 


Of course, most people were simply captivated by the notion of impacting the moon. Within the project we 
would joke that we were simply landing on the moon. . .with vigor, while the popular media routinely said 
we were bombing the moon. This view was further-fueled when the NASA-ARC Center Director blogged 
that LCROSS was a precision-guided bombing run of the moon. LCROSS was atypical in a number of 
ways, but the prominent feature of LCROSS with its dramatic “decommission-ing” in the bottom of a 
permanently shadowed crater on the moon, and this was the principal driver of media attention. 


It is not surprising then that the bulk of the attention to LCROSS focused on the time just before and after 
lunar impact. The popular media got very involved with their own spins, but even more interesting was 
the addition of the blogosphere, Facebook, Twitter and other online venues in which freedom of speech 
really took off. The multidimensionalism of this experience was at times surreal. In fact, it took on a 
rather serious tone when I started getting calls on my office phone shortly before lunar impact. 


On Oct 2, 2009, at 2:30 pm, a week before LCROSS lunar impact, I received a phone call at my desk 
from a woman who stated, “I want to talk to someone about NASA s bombing of the Moon and was told 
you are the right person. ’’ My eyebrow went up; after hearing some of what she said, I requested her 
name and contact number for forwarding along to NASA-ARC ’s Public Affairs Office to handle. I later 
learned that I was not the only one to receive such phone calls. 
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Figure 21-1: NOVA Science NOW coverage of LCROSS, the “Moon 

Smasher”, illustrating increasing interest in the mission. 


To add to the potential concern, not 20 minutes 

after that first call, I receive a second one from | was immediately SUSpicioUS about 
someone statins he was from the NASA-ARC 

„ ,.. TT . * ... , . , „„ a stranger expressing urgency in 

Credit Union with a gut basket and asking, How . r r »» 

can I get this package to the LCROSS flight team; ® 8"^ basket to the folks Vvho 

I need to do this right away. ” Now you can bet are flying LCROSS... //»*» mission 

that the earlier phone call put me into a certain that H>(iS going tO bomb the ttlOOU. 

frame of mind and so I was immediately 

suspicious about a stranger expressing urgency in getting a gift basket to the folks who are flying 
LCROSS. ..the mission that was going to bomb the moon. I later learned that the second call was 
completely legit and the credit union caller knew one of the LCROSS flight controllers, had heard about 
the team’s hard work, and wanted to send well-wishes to the team. The gift was a fresh-fruit basket and 
this is explained the urgency of getting it to them - before it spoiled. 
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www.nasa.gov Please bring your own Blankets, Chairs, and Snacks. 


National Aeronautics and Space Administration 


LCROSS 


(Venue officially opens at 5:30pm) 


Witness real-time space exploration as the Lunar Crater 
Observation and Sensing Satellite (LCROSS) 
impacts the moon to search for water ice and vapor. 


NASA Ames Research Center 

Exit Hwy 101 at NASA Parkway. Moffett Field, CA 


Thursday Evening through early 
Friday Morning, October 8th & 9th 


6:30pm - 8:30pm Thursday 

Presentations: S. Pete Worden, NASA Ames Center Director 
Charlie Duke, Apollo 16 Astronaut 
Victoria P. Friedensen. Advanced Capabilities. 
Exploration Systems Mission Directorate NASA 


8:30pm - 3:30am Thursday - Friday 

Outdoor Movies: Three Major Motion Pictures Shown 


Leam more about LCROSS at: www.nasa.gov/1cross 


3:30am - 7:00am Friday 

Lunar Impaci: Live Video Coverage 

Expert NASA Commentary 


open to the Public 

Respond early as seating is limited 

FREE ADMISSION 

Advance Tickets Required 

For outdoor movies 

To receive your free advance ticket(s) you must visit 
http://1cross.arc nasa.gov/impactnigfrt 


Figure 21-2: LCROSS Impact Night flyer from NASA-ARC. 
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These calls, however, prompted NASA Security to request an official report from me so they would have 
a record of the incident, including name and contact info, in case something bigger came of it. The reason 
for the special interest was that NASA-ARC was about to hold a live public event called, LCROSS 
Impact Night, to be held the night of the LCROSS impact. Thousands of people were expected to attend, 
congregating outdoors on a large lawn area, watching streaming data come down from the LCROSS 
spacecraft and projected on a giant screen. Documenting any and all calls that could reveal a potential 
threat to disrupt that public event was important. 

NASA decided that it should proactively get in front of what could become a watershed level of 
apprehension by some in the public, and crafted a statement that it sent to all parties who expressed 
concern about the LCROSS mission: 

“Thank you for your interest and concern about the LCROSS mission. In light of your interest, 
we would like to provide you with further information about the mission. NASA wants to make 
dear that there are no explosives on the Centaur that LCROSS is directing to the Moon. 
NASA has spent months since launch making sure that all possible fuel is emptied from the 
Centaur. The impact will cause a small crater, smaller than a tennis court, to form in a pre- 
existing crater. The location of the impact will be clearly marked on maps of the lunar surface 
so that future lunar explorers would be able to visit the impact site safely. 

Beyond the phone calls and some of the very genuine 
concern being expressed from some quarters of the 
public, LCROSS had also become the subject of 
several fringe viewpoints that flourished in the days 
prior to impact. Early in the project, I saw claims that 
LCROSS was an American imperialistic effort to own 
the Moon, with headlines that read, “[President] Bush Bombs the Moon.” Others saw the LCROSS 
mission as a symbolic muscle-flexing toward China in response to their own lunar and military ambitions. 
I even read that the LCROSS mission was a secret attempt to test an anti-matter weapon on the moon 
where the US could best hide such activities! My favorite read was a website claiming that LCROSS’ real 
purpose was to massacre a secret alien civilization on the moon, and the permanently-shadowed crater 
LCROSS was targeting was precisely where they lived. This group of individuals then took things a step 
further and attempted to mobilize people to put a stop to the impending massacre. At times, this lunacy 
was entertaining and more fantastic than a Hollywood script. Some examples from the blogosphere: 


My favorite read was a website 
claiming that IX ’ROSS' real 
purpose was to massacre a secret 
alien civilization on the moon. 
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From: (removed)@sbcglobal.net 
To: (removed)@yahoogroups.com 
Sent: Thu, Sep 17, 2009 6:16 pm 

Subject: Oct. 8 Joint Psychic Exercise: stop the Cabal's crashing NASA's LCROSS and Centaur rocket 
into a Star Visitor colony within Cabeus A Crater 

Friends, 

Two months ago on July 21 I sent out a message that the NASA LCROSS satellite heading for the Moon 
was a Cabal project to annihilate a Star Visitor colony living in a crater near the Moon's South Pole. Now 
there is new additional information which requires us to prepare to move into action at the right time. 

To remind: the Cabal operation involves crashing NASA's LCROSS satellite and its accompanying 
Centaur rocket stage into the Moon's surface within a shadowed South Pole crater, Cabeus A, on October 
9, 11:30 UT (07:30 am EDT, 04:30 am PDT). See attached photo of Cabeus A Crater. The Star Visitors 
colony is located in the upper northwest quadrant, up against the crater wall. 

The Cabal within NASA know that there is a colony of Star Visitors living within Cabeus A Crater. The 
Cabal's secret objective is to use the LCROSS and attached rocket stage to obliterate the Star Visitor 
settlement residing within that crater. 

I attempted to relay this warning message to President Obama, and to Vice-President Biden (who 
normally oversees the government's Star Visitors programs) through a channel available to me. But 
unfortunately the Vice-President is in the Middle East, and Cabal asset Hillary Clinton, Secretary of State, 
intercepted the message and saw to it that it did not reach effective levels in the White House. 
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This call to arms continued for several pages, making references to UN Charters and the Geneva 
Convention to argue against the LCROSS mission and asked for a Joint Psychic Exercise to redirect 
LCROSS and the Centaur rocket away from the Moon. Another blog carried a similar thread: 


NASA moon bombing violates space law & may cause conflict with lunar ET/UFO civilizations - June 19, 
4:14 PM 

The planned October 9, 2009 bombing of the moon by a NASA orbiter that will bomb the moon with a 2- 
ton kinetic weapon to create a 5 mile wide deep crater as an alleged water-seeking and lunar colonization 
experiment, is contrary to space law prohibiting environmental modification of celestial bodies. The NASA 
moon bombing, a component of the LCROSS mission, may also trigger conflict with known extraterrestrial 
civilizations on the moon as reported on the moon in witnessed statements by U.S. astronauts Buzz Aldrin 
and Neil Armstrong, and in witnessed statements to NSA (National Security Agency) photos and 
documents regarding an extraterrestrial base on the dark side of the moon. 


1 The blog characterizes the U.S. aggression toward the “extraterrestrial civilizations and settlements on 
the moon” as war crimes. It further states that U.S. astronauts, NASA employees, Soviet scientists, and 
the NSA have confirmed extraterrestrial presence on the moon. This of course includes what is called 
conclusive evidence of an alien moon base on the far side of the moon, and eyewitness sightings of UFOs 
on the moon shortly after the Apollo 11 Moon landing on 21 July 1969. 

Other articles focused less on inhabitants of the moon and more on concern for the moon itself, offering 
graphic imagery to convey a shock-and-awe quality to it all. One image included the statement, “ There is 
also the potential risk of another alien invasion of sorts and all out infestation of lunar bacteria raining 
down on the Earth. ” 1 2 
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Other threads argued conspiracy theories, stating that the new Obama administration was about to come 
clean after years of hiding this information from the general public: 

“An official announcement by the Obama administration disclosing the reality of extraterrestrial 
life is imminent. For several months, senior administration officials have been quietly deliberating 
behind dosed doors how much to disclose to the world about extraterrestrial life. Dissatisfaction 
among powerful institutions such as the U. S. Navy over the decades-long secrecy policy has given 
a boost to efforts to disclose the reality of extraterrestrial life and technology. 

Obama's September 24, 2009 chairing of the UN Security Council meeting on nuclear non- 
proliferation and disarmament, signaled his emerging leadership role in tackling major global issues 
such as nuclear weapons. The Nobel Peace Prize was an important step in giving global legitimacy 
to President Obama in making an extraterrestrial disclosure announcement. Obama is therefore 
poised to play a prominent role in the increased global governance that will be necessary after an 
extraterrestrial disclosure announcement. The timing would most likely coincide sometime soon 
after his Nobel Peace Prize acceptance speech on December 10, 2009 in Oslo, Norway. ” 3 

Finally, there were articles that were not so much about civilizations under attack by the US, but 
supposed militaristic applications of LCROSS: 

“Of course, there is much more behind this attack than casual scientific curiosity on whether or 
not there is water on the Moon. First of all, since the long-range accuracy of intercontinental 
ballistic missiles has never been proven to work, the LCROSS suicide mission serves as a live- 
fire test exercise for US war strategists with an interest in the precision of orbiting satellite 
weapons - in other words, the southern hemisphere of the Moon will be turned into a firing 
range, making this mission one giant leap for the global reach of space warfare. Secondly, 
LCROSS has been promoted as "the vanguard" for the US military-industrial-entertainment 
complex's return to the Moon - according to NASA, finding water is a necessary first step for 
"building a long-term and sustainable human presence" there. Historically, the purpose of 
exploration has always been the exploitation of resources and the colonization of territory 
without regard for ecosystems or indigenous peoples, and clearly the Moon is the next territory 
coveted by imperialists. ” 4 

As one of the original participants in the 
development of the LCROSS proposal, I of 
course knew that none of these angles were 
accurate, but that didn’t mean NASA could 
ignore them. Managing this lunacy required 
carefully combing through the various 
claims to determine who was credible and therefore worthy of contacting to assure their facts were 
straight, and who was “fringe” and not worthy of official NASA engagement. Public discussions with 
questionable sources can draw you into the silliness and potentially get you trapped defending the truth 
against the absurd. This is where you need to keep the public affairs office close by, and your media 
training at the ready to navigate the right course. Did I mention you should get media training? It is an 
absolute requirement, to prepare for discussions such as these. 


Public discussions with questionable 
sources can draw you into the silliness 
and potentially get you trapped 
defending the truth against the absurd. 
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Finally, some discussions in the blogosphere didn’t specifically have an agenda, but simply got the facts 
wrong: 

“The probe [spacecraft instruments], however, has even more up its sleeve. Microwave ovens 
work by exciting water molecules by emitting electromagnetic waves at the frequency at which 
water resonates. LCROSS is also armed with a microwave emitter and it was hoped that 
energizing molecules of water would make them easier to detect. 5 

With all the swirl of interest and excitement over this mission, we found it entertaining to see how 
different authors would sound-bite the LCROSS mission or its activities. Below is a sampling of some of 
my favorite LCROSS -related headlines: 

♦ “NASA to Bomb the Moon Friday ”, with a reference in the article to NASA firing a “bomb-laden 

missile” 6 

♦ “NASA punches Moon in the FACE ” 7 

♦ “NASA s moon blast called a smashing success ” 8 

♦ “NASA craft smacks the moon in quest for water” 9 

♦ “NASA s Big Flit: LCROSS Impacts On The Moon ” 10 

♦ “The moon is due for a double whammy ” 1 1 

♦ “NASA Takes Aim at Moon with Double Sledgehammer ” 12 

♦ “NASA rocket on crash course with moon ” 13 

♦ “You can watch NASA give the moon a one-two punch ” 14 

♦ “Moonstruck - Making one giant thud for mankind - NASA Lunar Mission Combines Romance Of 
Moon With Lust For Cosmic Violence” 15 

♦ “Take that, moon!” 16 

♦ “NASA Set to Dive Bomb the Moon ” 17 

♦ “KAPOW! NASA Smacks the Moon in Search for Water Ice ” 18 

The LCROSS mission even became the darling of the late-night and cable television comedy 
monologues, but it wasn’t all favorable to LCROSS. I soon came to realize I needed a sense of humor 
about these things; besides, this kind of press makes NASA and its missions more approachable to the 
general public. Some examples: 

♦ MSNBC Rachel Maddow Show “It’s Time to Bomb the Moon. Seriously, we are going to bomb the 
moon. ...Other than the fact that this sounds like a heck of a lot of fun for the NASA folks who get to 
literally shoot the moon, why are we doing this? Well, apparently it’s all about the debris cloud... ” 19 
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♦ CBS Late Show with David Letterman 

“Now, listen to this. I’m no rocket scientist so 
far be it from me to tell these people who are 
rocket scientists how to do their business, but 
NASA, they’re shooting a missile. ...They’re 
going to launch a huge missile - kaboom - right at the moon, looking for water. And I said, ‘Why 
not? Now that everything here is taken care of on Earth, why not? We ’ve got no problems here. Let s 
just go give it a shot. ’ ” 20 


The LC’ROSS mission even became 
the darling of the late-night and 
cable television comedy monologues. 


♦ CBS Late Show with David Letterman “Letterman s top 10 list: “ Top Ten Signs The Head of 
NASA Is Nuts.” 21 

♦ CBS The Late Late Show with Craig Ferguson, Ferguson jokes, “At NASA, the countdown is on. 
After years of wasting taxpayer money on research to increase the quality of life here on Earth and 
all that rubbish, NASA is finally doing something cool. They 're blowing up the moon!.. .NASA says 
it s just an experiment. They say they will spectro-analyze the debris cloud, looking for water on the 
moon. I think we all know what the real mission is - reminding the moon that it’s still our bitch. ” 22 

♦ NBC Late Night with Jimmy Fallon, Fallon jokes, “NASA announced that it will crash a rocket 
into the moon this Friday morning, hoping to discover water there. To make sure the crash is hard 
enough, it’s being piloted by the Detroit Tigers. So what NASA is doing, they’re crashing a rocket, 
which will have the energy of two tons of TNT. It ’s part of NASA s new strategy, ‘What would Wile E. 
Coyote do?’” 23 

♦ CBS Late Show with David Letterman, Letterman joked, “NASA is going to launch a rocket to 
the moon on Friday. They’re going to shoot a rocket to the moon. Just going to - kaboom, kaboom! 
...The government says don ’t worry, that they ’re pretty certain we will be greeted as liberators [Iraq 
war reference], ...And you know that is how we do stuff. We launch the attack then we look for the 
evidence. ” 24 


♦ ABC’s Jimmy Kimmel Live, Kimmel joked, “The United States is bombing the moon tonight. NASA 
is honestly planning to fire a rocket-powered explosive into one of the lunar poles. See, this is what 
happens when your president’s slogan is ‘Yes we can. ’ ...If I was NASA, I would have auctioned the 
chance to fire the missile that blows up the moon on eBay, right? I mean, it would have paid for itself. 
“Dude, who gets to push the button to explode the moon? Me, that’s right. ’ ... What a waste. ” 25 

♦ NBC’s Late Night with Jimmy Fallon, “I was reading that NASA is going to fire that rocket into 
the crater of the moon tomorrow morning, and people can follow the mission on Facebook and on 
Twitter. And you can go to Friends ter, too, and follow the original moon landing. ...Speaking of 
NASA, you guys heard about this asteroid that can strike the Earth in 2036? ...NASA just 
downgraded the threat collision to 1 in 250, 000. . . . That means you have a better shot at getting 
crushed by an asteroid than winning the grand prize of McDonald ’s ‘Monopoly ’. ” 26 

♦ NBC’s Jay Leno Show, Leno joked, “Well, here’s something interesting. Tomorrow, NASA scientists 
will crash two spacecraft into the surface of the moon in an effort to find ice... . The spacecrafts are 
named Amtrak One’ and ‘Amtrak Two. ’I’ll tell you, scientists are very excited about the possibility of 
ice on the moon. Not as excited as personal injury attorneys, but almost as excited. ” 27 
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♦ Comedy Central’s Daily Show with Jon Stewart, Jon Stewart joked, “There seems to be a general 
consensus between the President’s mortal enemies, and the Taliban, that he hasn ’t done enough to 
deserve this award. Time to prove 'em wrong. Mr. President, you ’ve just gotten the Nobel Peace 
Prize. What game changing, peace loving move will you make, just 90 minutes after finding out the 
news, to show that they gave the Peace Prize to the right man? ‘NASA literally bombed the moon. ’ 
Prize for peace on Earth, baby. Suck that, moon!” 28 

LCROSS also provided a surprising amount of 
fodder to the political cartoon circuit. These 
cartoons either directly addressed the LCROSS 
mission, or the context in which it existed. 

Some were just plain fun while others played 
off of the “bombing the moon” frenzy that took 
hold in some circles prior to LCROSS impact. 

Others still latched on to the interesting timing 
of the announcement that President Obama had 
received the 2009 Nobel Peace Prize against 
this “bombing” backdrop - the announcement 
being made just 90 minutes before LCROSS 
was to impact the moon. Here are some 
examples of many that appeared: 
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BREWSTER ROCKIT: SPACE GUY! By Tim Rickard 
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One thing is for sure: LCROSS garnered an extraordinary 
amount of attention. Most of the excitement occurred 
during the final act of the mission. Whether for excitement 
surrounding the “bombing,” answering the water question, 
or more fringe interest, LCROSS drew big numbers during 
the impact event. In fact, almost 392,000 people watched 
the LCROSS impact video live on www.NASA.gov with 
5.4 million visitors simultaneously looking at the www.NASA.gov web site, the second-largest online 
event in NASA’s history. The event also set a new bandwidth record with data transfers of over 75.5 
gigabits per second spread among live video and the web site. All told, during the LCROSS impact 
event, we moved over 85 terabytes of data, or about the equivalent of nearly 130,000 CD’s. 29 


5.4 million visitors 
simultaneously looking at 
the www.NASA.gov web site, 
the second-largest online 
event in NASA’s history. 



These numbers made LCROSS the fifth most popular live Internet event in the history of the Web 30 , and 
#1 on Yahoo.com for search popularity 31 . 
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Figure 21-4: LCROSS Impact night Yahoo top searches: LCROSS is #1. 

As mentioned earlier, the morning of LCROSS impact, ARC 
held a giant outdoor, all-night party in which music, food, 
movies and live LCROSS coverage were provided so that the 
general public could gather and participate in the excitement. 

People signed up online to get reserved passes and could stay 
all night, even bringing tents. We were upstairs in the mission 
ops control room, while a big party went on, with thousands of 
people and our families outside on the NASA-ARC Moffett Field parade grounds. About nine hours 
before impact, LCROSS headed into the moon, releasing the Centaur. We watched it fade away from 
view as it accelerated into the moon, ultimately impacting Cabeus Crater, followed four minutes later by 
the shepherding spacecraft impact. 

Of course the moon did not explode, and the destruction some had feared never materialized. In fact, the 
mainstream media and blogosphere wondered where the big explosion was. Did LCROSS miss the moon? 
What a rip-off! I got my kids up to see this and for what?! There was no obvious impact in the video that 
was shown on TV and on the web. This non-event got so contentious that some people accused NASA of 
over-hyping the whole event to draw attention to it. I remember doing some satellite press after shortly 
impact; I was pulled away by my public affairs person telling me I had to conduct an urgent phone 
interview. One of the space industry magazines was openly accusing NASA of over-hyping the event and 
threatened to write a story to that effect unless they got me on the phone within fifteen minutes.... 


We were upstairs in the 
mission ops control room, 
while a big party went on, 
with thousands of people 
and our families outside. 
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While you do not want to play into such tactics, I needed to speak with this individual. He was from a 
notable aerospace media outlet and had been following the mission and interviewing me along the way. I 
was able to explain to him that this less dramatic result yielded valuable scientific data about the nature 
of the constituents in the bottom of these unknown craters, and we needed time to analyze and study the 
data. It was true; if we struck u nkn own rocks in the bottom of the crater we’d likely have an energetic 
impact with debris, however, if the bottom of the crater was filled with talc-like dust, the energy of the 
impact would be slowly absorbed as thermal energy and not be much of a visible spectacle. I was able to 
talk the interviewer away from over-reacting 

and was an extraordinary learning moment for Olli* of the spaCt* industry mUgUZincS 
me. Much of the perception of the mission in WUS Openly UCCUSing NASA of 

the industry media was hanging on how that over-hyping the event and threatened 
follow-up discussion was handled. Does this . . 

, , F „ to write a storv to that effect. 

jeet like a Class D context r 


In a matter of moments LCROSS had gone from having to assure people that we were not going to hurt 
the moon (or any claimed inhabitants there), to asserting the fact that we did actually hit the moon. We 
held a press conference within a couple hours of impact, but this was not enough time to collect the data 
in a meaningful way, so we were limited in what we could show the public. We could not yet answer the 


186 



Chapter 21 — Managing the Lunacy 


water question without being scientifically reckless. Worse than keeping the public waiting is releasing 
premature information and then having to retract it. The data needed to be poured through very carefully. 

So why wasn ’t there a big explosion ? Over the course of the project, LCROSS performed numerous tests 
at the Ames Vertical Gun Range (AVGR) at NASA-ARC to specifically study how the LCROSS impact 
would look, both to the LCROSS spacecraft, and from Earth. The AVGR is actually a ballistic test range 
where a projectile can be shot at different speeds, through a tube (like a giant muzzle of a gun), into an 
evacuated chamber (no air - just like the moon), impacting the soil of your choice; in our case it was 
simulated lunar regolith. The impacts of cylindrical shapes at different velocities, at different impact 
angles to the surface, and with different impact orientations were all analyzed based on the current 
understanding of lunar regolith from the equatorial regions of the moon from the Apollo missions. Once 
we poured through the actual LCROSS impact data, we did see a plume from multiple cameras and from 
multiple missions (LRO and ground assets did witness different parts of the impact plume and crater in 
some cases), but it was not as extravagant as many had hoped (or feared). 



Figure 21-6: Impact Night artwork illustrating that you might be able to see the impact plume. 
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Figure 21-7: NASA-ARC Ames Vertical Gun Range (AVGR) 

facility impact testing (Images courtesy of Peter Shultz). 


The LCROSS science team found that the regolith at the location of impact is highly unconsolidated and 
best described as being of a talc-like consistency. Recall that the nature of the regolith in the bottom of 
the permanently-shadowed craters was u nkn own since no instruments, or astronauts, had ever been there. 
Because of the non-rocky, fine-powder nature of the impact regolith, the bulk of the impact energy went 
into thennal heating of the regolith instead of into kinetic energy of rocks and debris, which would shoot- 
away from the crater floor. The local thermal heating of the lunar local soil then vaporized the local 
volatiles in the regolith, including the later-discovered water. This led to a very clear vapor cloud, which 
formed above the crater floor, and was witnessed from various assets. 


The speed of media and the blogosphere is, however, 
orders of magnitude greater than the speed of 
scientific rigor, which is intolerant of retractions 
following a rushed conclusion. Any credible science 
team takes its findings and conclusions before an 
independent peer review to enable independent, 


A rigorous process takes time, and 
with the excitement surrounding 
the culminating LC ROSS event, 
the public was left with the 
question, well.,. was there water ? 


critical eyes to look at the conclusions being made - 
this is a very normal, rigorous process in evaluating scientific discoveries. But this rigorous process takes 
time, and with the excitement surrounding the culminating LCROSS event, the public was left with the 
question, well... was there water? This left an information vacuum, which the media and blogosphere 
will fill if the project team does not. For a week or two, the media pretty much waited for the results, but 
when the water results didn’t come until a month later, some in the media started making their own 
conclusions to fill the void. Enter the fringe elements again: 


Friends, 


I will briefly share what I "saw" during my participation in yesterday's successful Joint Psychic 
Exercise (JPE), and then share what Star Person Incarnate Human Fran has to say about the 
Cabal-controlled media and the LCROSS NASA mission. I moved over in time some minutes, 
and moved (astrally) over in space a certain distance and then, operating from "there", went 
out psychically to LCROSS and Centaur booster as they were streaming towards the Moon. 
Next I enwrapped LCROSS in a telekinetic force and redirected it onto a course to the left so 
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it was aiming towards one Moon-diameter's width left of the Moon's left side. Then the same 
was done with the Centaur booster rocket. After LCROSS and Centaur were speeding on their 
new course, I engaged first one, then the other, with strong dissolution energy to unbind the 
Strong Force bonds holding their atoms together as molecules. Moving from top to bottom, I 
un-did the Strong Force bonds, causing the component materials of these space vehicles to 
come apart at the molecular level. This process also safely dismantled the advanced munitions 
which were secretly aboard these space vehicles. Nothing left but a expanding array of atoms. 
Mission accomplished! Thus the star folks lunar colony within Cabeus A Crater is safe from 
overhead bombardment. Hoo-rah!" 22 


Others in the fringe community did not claim to mentally vaporize the Centaur and LCROSS spacecraft, 
but alleged that NASA had already discovered an ancient base at the moon’s South Pole following earlier 
data from the LCROSS mission, and the reason nobody saw the LCROSS debris plume was because “the 
probes struck a building which swallowed the effects of the explosion.” While these absurd accounts are 
easily ignored, less sensational media were still very hungry to talk to anyone about the mission, and it 
became apparent to me that the team needed to make sure it clearly understood the differences between 
expressing personal opinion and being a representative of the project. Here is an email I sent to the team: 


From: Daniel Andrews 
To: LCROSS Team 
Sent: Oct 22, 2009 

LCROSS team- 

LCROSS has certainly garnered a fair amount of attention the final days of the mission and in discussions 
since. It is understandable given the nature of this mission having a final culminating event. ..the media 
frenzy surrounding “bombs” and “explosions” certainly brought additional attention which is filling the 
blogosphere. 

On this topic, I want to put forward a request for LCROSS team members in their participation in public 
commentary and discussion forums. I cannot and do not want to limit anyone's ability to comment on the 
mission, but only as individuals with individual points of view. Representing yourself as a project person 
carries implied authority which requires project participation. 

The blogosphere is filled with opinions, critics, fans, and everything in between about any subject you can 
imagine - LCROSS is no different; but the moment that a “project member” weighs-in, there is implied 
authority and representation of a project position. The project position is not owned by individuals, but by 
the project and the Agency. I have seen project personnel discussions on NASAWatch and other 
locations which were surprising to me. 

One final variant: There are some of us whose names are automatically associated with the project, even 
if we never identify our affiliation. For example if I started weighing-in on a topic (no matter the opinion), it 
is implied that this is a project position. There are others on this team which fall into the same category. 
These folks have a tough time being anonymous and I'd recommend coming to see me if there is a topic 
in which I should be aware. 

So please be careful out there. 

Thanks 

-D 
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Figure 21 -8: Never-published test cover of Sky & 

Telescope magazine, prior to release of LCROSS results. 

As the days rolled on after the impact, the tone was trending negative, specifically because there was no 
official NASA explanation of what happened. Even some notable scientific magazines started floating 
unpublished, trial magazine covers illustrating their own conclusions - later to be unfounded. I had many 
discussions with the LCROSS science team and the Public 
Affairs Office (PAO) that we need to fill the void out there, or 
the media and public would fill it for us. This tension required 
careful management of messaging to the public, while 
preserving scientific integrity. . .a tough constraint. 


LC ROSS began to turn 
previous conceptions about 
the moon on their head. 
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LCROSS science ruled at the end of the day, having not only shown that there was indeed water ice down 
in the bottom of those permanently-shadowed craters, but there many other very interesting volatiles 
there as well, and strong evidence that these craters had been formed by cometary impact. LCROSS 
began to turn previous conceptions about the moon on their head. We held a press conference sharing this 
information and the media frenzy was amazing. This lunar discovery appeared on the cover of the New 
York Times, the Washington Post, the Wall Street Journal, USA Today, the Los Angeles Times, the San 
Francisco Chronicle, and untold other papers. 

Social media sites also went crazy, with the water-on-the-moon press conference arriving on the Twitter 
top- 10 trends in well under an hour with NASA at #3 and LCROSS at #7. The news conference brought in 
12 different media outlets to NASA-ARC with 26 reporters calling in. LCROSS became the lead story on 
CNN and was placed on the front page of CNN.com. It also made the top 10 trends on twitter and stayed 
there for more than 2 hours. LCROSS was the lead story on MSNBC, Reuters, the Wall St. Journal and 
Washington Post web pages. Google even commemorated LCROSS ’s discovery with a dedicated Google 
logo. The briefing itself aired live on German TV, with translation. Multiple post-briefing follow-up 
interviews included CBS, NPR, ABC, and newspapers in Brazil and Canada. Magazines and journals also 
caught up, with LCROSS making Rolling Stone, and Time Magazine’s “The Decade From Hell” issue 
with NASA’s discovery of water on the moon being cast as one of the few bright spots of the decade. 

As before, the internet became captivated by the discovery. We started to see amusing imagery dedicated 
to the finding of water on the moon. 
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Figure 21-9: Google search page skin the night of the LCROSS lunar impact. 


Surprises even turned up on some of the auction websites like eBay, where we found team memorabilia 
for sale... . No, that is not my eBay account name, but it is a project-internal team patch. LCROSS also 
created new fodder for discussion on everything from Lunar-mining rights, to historical preservation of 
notable lunar sites: 

LCROSS Impact Site Seen As Archaeological Site: Dr. David Livingston at The Space Show 
interviewed Ann Darrin of the Johns Hopkins University Applied Physics Laboratory and Beth 
O'Leary, an anthropologist at New Mexico State University, who just wrote the "Handbook Of 
Space Engineering, Archaeology, and Heritage. " The show focused on space archaeology and, the 
legal regime that would be necessary to preserve sites in space and on a celestial body and the 
guests said the LCROSS impact site was now a new archaeological heritage site, and there was 
discussion about the preservation value of such a site . 33 

When you return to the origin of the LCROSS mission, however, it was a NASA mission that served 
NASA’s exploration purpose. This relevance to NASA was exemplified at a dinner address that NASA 
Administrator Charlie Bolden gave at the October 21, 2009 Wernher Von Braun Memorial Symposium. 
Here is a snippet from his speech: 
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<snip> ...Another reason that this was an exciting few 
weeks for space is that we just conducted the impact 
of the LCROSS lunar satellite and its rocket's upper 
stage into the Moon. Now, many of you will have heard 
about the recent news that one of NASA 's 
instruments, onboard an Indian space probe, detected 
quantities of water molecules distributed across the 
lunar surface. LCROSS - which stands for the Lunar 
Crater Observation and Sensing Satellite - was meant to follow up on those significant 
observations by aiming for a permanently shadowed crater of the moon's South Pole, where some 
scientists postulate that there could be deposits of water ice. This is exciting science. Whether 
or not we find signs of water ice, I believe that LCROSS is the kind of mission of which NASA 
needs to do more . 34 

I can’t think of a more relevant or satisfying acknowledgement of the importance of the LCROSS 

mission. 


*7 believe that LCROSS is 
the hind of mission of which 
NASA needs to do more. ” 
NASA Administrator 



yeak, we ki-f tkat 
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Figure 21-11: Actual eBay auction selling an LCROSS mission patch. 
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When LCROSS was selected, Mike Griffin was the NASA Administrator, and one of his goals during 
his time was to realign the Agency’s activities with a pragmatic and business-like atmosphere for running 
projects. In February 1993, Griffin sent an email to a huge, all-Agency staff distribution with the subject 
“Costs of Doing Business.” 1 In the email he explained how his 
experience in working with industry had shaped a way for the 
Government to most efficiently work with industry to 
accomplish Agency goals cost-effectively. I thought it would 
be interesting to see how the innovative and maybe atypical 
approaches that LCROSS employed compared to the vision of 
the NASA Administrator. 

Below is an exact transcript of this email in italics, interleaved with my assessment of how the 
management of LCROSS compares to his experience: 


All cost reductions 
ultimately imply doing a 
given job with fewer people. 
- Charlie Bolden 


Sent: 12 February 1993 
Subject: Costs of Doing Business 

Below are some points regarding the cost of doing business in NASA. These have been gleaned from 
personal experience, and from numerous interviews with industry on the specific topic of how the 
Government can work with industry to produce a guality product at lower cost. 

All cost reductions ultimately imply doing a given job with fewer people. There are in truth no materials 
costs, tooling costs, etc. There are only labor costs. So we need to manage in such a way as not to cause 
or allow the contractor to put non-value-added people on the job, either to make money for himself, or to 
answer the guestions of Government micromanagers. 



This is already in good alignment with how the LCROSS project worked, both with the Government staff 
and with Northrop-Grumman staff. Both staffs were notably smaller than for typical missions. While this 
was important across the project, the expense of the contractor staff in particular, absolutely required this. 
There was no room to park people on the project. People came onto the project to do work and then exited 
the project when their work was completed. It’s not too dissimilar from homebuilding: once a site is 
prepared for building, a crew comes in and pours all the foundations and then leaves. Another crew then 
does all the framing, and leaves; another crew hangs all the doors and windows and then leaves, etc. This 
ensures that the project pays only for the time required. Some contractor firms are designed to support this 
approach while others are not; the same applies to the Government. It takes strong management and a 
supportive, healthy infrastructure to enable people to roll on and off projects in this way. 




Cost models are a significant part of the problem. By definition, they fit a curve to a set of historical data, with 
heaviest weighting for the worst offenders (i.e., biggest overruns), in an attempt to ensure that the next 
project will not overrun. This rapidly becomes a self-fulfilling prophecy. 
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During the proposal phase, LCROSS had the Aerospace Corporation (Aerospace) conduct an 
independent cost estimate (ICE) on LCROSS. The estimate did indeed come in quite high when first run, 
as it was indeed comparing to missions which were doing things in traditional ways, growing the 
resultant estimate. The team had a follow-up dialogue with Aerospace, wherein we explained our 
considerably more lightweight approach and extraordinary use of proven and commercial products. We 
had to explain where the assumptions put into the Aerospace ICE did not hold for us. Aerospace then 
perfonned a revised estimate and compared the LCROSS approach to smaller missions of similar 
complexity - a strong driver of project cost. That revised ICE came in very close to that of the LCROSS 
proposal. . .and as history has shown, in alignment with our actual costs. 




Be more careful in Phase A and B; better requirements definition reduces change traffic and hence runout 
cost. Phase A should focus on requirements only, and should be done in-house; solutions developed in 
Phase A should have an existence-proof flavor. Disallow top-level requirements changes after Phase A. 



Since LCROSS was a mission of opportunity, the proposal development phase was effectively “Phase A” 
for project development. Upon selection, LCROSS entered Phase B development and had a Systems 
Requirements Review (SRR) just a couple months after authority to proceed (ATP). Our approach, 
nevertheless, matches the Administrator’s prescription. The LCROSS proposal was very detailed on how 
the mission would be accomplished and when we subsequently prepared the requirements for the SRR, they 
flowed directly from the foundational approach of the proposal. We did not enhance; we flowed down. 


Government should specify performance requirements, and let contractors innovate in Phase B. Disallow the 
overwhelming tendency of the Government to specify the details of the solution, unless Phase C/D is being 
done by the Government. 


LCROSS took the approach of developing Level 3 requirements that flowed down from the proposal. The 
program office and Exploration Mission Directorate then flowed up requirements from these Level-3s to 
make sure they honored the original proposal concept and did not steer it in a different direction. Prom the 
Level-3 requirements, Level-4 requirements and below were developed by the appropriate providers of 
those requirements. In the case of the spacecraft, Northrop-Grumman wrote Level-4, Level-5, and so on 
requirements based on their experience for satisfying the project/ Government Level-3 requirements. While 
the LCROSS project had full insight and access to those subordinate requirements, we did not steer them. 
We allowed both the development and later verification of those requirements to be completely owned by 
the contractor. The LCROSS project simply monitored the satisfaction of the Level-3 requirements. 


This approach also held for Government-provided products. The LCROSS payload suite was developed 
in-house at NASA-ARC. The Level-3 requirements defined the perfonnance of the LCROSS payload 
and then the Government team working the payloads developed Level-4, Level-5, etc., requirements 
from the Level-3 s. The project only cared about satisfying the Level-3 requirements and allowed the 
provider (in this case the Government) to develop their own subordinate requirements. 
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LCROSS was selected as a cost-capped mission. We had a set of key performance parameters (KPPs), 
which were the basis for possible termination of the project. One of the four LCROSS KPPs was for total 
lifecycle cost ($79M) and another was for schedule (matching LRO’s launch schedule). If either of these 
were not met, LCROSS could be cancelled or flown partially incomplete as a mere launch vehicle 
adaptor for LRO in order not to impact the LRO launch date. These were compelling drivers for 
LCROSS and so our life cycle cost estimate was continually updated and monitored. We consciously 
made decisions that ultimately played to cost containment. 



Emphasize “hardware rich” approach to doing programs; i.e., early and continuing use of hardware test beds, 
prototypes, and flight tests. 



Because LCROSS was a cost-capped project, the degree to which we could be “hardware rich” was 
limited. In fact we could not build as many engineering test units as many programs might, specifically 

because of cost; however, we did try to make testbeds as rich 
as feasible. These included everything from the avionics 
“FlatSat,” where we attempted to identify functionality 
difficulties early in flight software development, to the use of 
the whole-spacecraft simulator. The latter was used in early 
operations training, and in flight; we’d both test software 
before we uploaded it to the spacecraft, and troubleshoot things that didn’t go as planned. NG also had 
rich simulator environments for different spacecraft systems, like battery simulators, spacecraft dynamic 
simulators, load simulators, etc., which enabled the project to test in advance of activities and 
troubleshoot problems after the fact. 



In areas where we are not trying to push the state of the art, encourage use of off-the-shelf and commercial 
hardware. Where we are trying to push the state of the art (a legitimate, even mandatory activity in NASA), 
we need to devote more effort to verifying that new technology is ready before including it in the larger arena 
of an ongoing flight project. Otherwise, delays in the new technology item pace the whole program, with cost 
escalating according to the marching army effect. 



Use of commercial hardware was a basic tenet of the LCROSS project. As mentioned in the Managing 
Resources chapter, we attempted to be a “glue mission” in which we took existing, proven hardware and 
glued it together to satisfy our project requirements. In the case of the payload instruments, where we used 
the most commercial (primarily non-aerospace) components, we had to ran them through suitability testing 


Use of commercial 
hardware was a basic tenet 
of the LCROSS project. 
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at NASA-ARC. Some of the components 
required multiple iterations on testing, as we 
discovered deficiencies that made them 
unsuitable for flight, but the net outcome was 
dramatic savings. The commercial vendors 
liked working with the LCROSS team, 

because of the symbiotic relationship we established with them. The Government would test the vendor’s 
commercial hardware and provide the results (free to them). The commercial vendors were very interested 
in improving their products to make them worthy of flight, so this was a win-win scenario. 


Concurrent teaming is always desirable 
because there are fewer hand-offs to 
others, reducing the likelihood of 
missing something in the process. 



Concurrent teaming was the way for LCROSS both because it was cost-efficient, and because the project 
timeline was sufficiently short that it was feasible for many on the team to work the project from the start to 
the end. Concurrent teaming is always desirable because there are fewer hand-offs to others, reducing the 
likelihood of missing something in the process. The difficulty the small team has with this approach is you 
cannot afford to cover the costs of parallel teams over the development lifecycle. This means many who 
would come to be operations crew were actually involved in the spacecraft development and then 
transitioned-over to operations, bringing with them their experience and knowledge of the spacecraft. 
Larger projects carry a parallel team of operations crew 

who follow development of the spacecraft so that they Without schedule discipline , there 

can ultimately fly it later. This is too costly an approach is no discipline in U program. 
for a small, cost-capped project, so we re-purposed _ |y| ( , nffin 

team members as they moved through the project cycle. 

For example, the NG team that built the spacecraft was clearly in the best position to understand the 
spacecraft’s performance on orbit. The project could not afford to have an operations team following 
the spacecraft development, so we struck a compromise: we stationed liaisons down with NG during the 
different phases of development to understand the spacecraft, and then those liaisons later joined the 
operations team. They helped considerably when it came to flying the spacecraft, but we knew it would 
not be sufficient, so we brought representative subject matter experts onto the operations team from NG. 
The contractors are more expensive, but the timeline is short and the labor spent somehow transferring 
NG spacecraft knowledge to another team is both expensive and ineffective; we had to work the cost/ 
effectiveness balance with our tight schedule. 


Reducing development time almost automatically reduces cost. Without schedule discipline, there is no 
discipline in a program. 


201 


Chapter 22 — Calibrating Our Approach 


The LCROSS project tried to keep the spacecraft and mission design as simple as possible in order not to 
require considerable time in the design phase. The idea of gluing together the mission helped considerably 
in this regard. Moving through the design phase quickly enabled more time to be spent in testing, although 
in a schedule-constrained mission like LCROSS, we were not able to dawdle in the testing phase either. 

The LCROSS project also tried to make sure that the integration flow was as flexible as possible in order 
to accommodate unexpected changes in the schedule of avionics deliveries. The flower-petal design of 
the LCROSS spacecraft, including the NASA-provided payload panel, made this relatively easy. The 
integrated master schedule was the focus of the project activities, with schedule slack managed as a 
resource throughout. One team’s schedule slack was used to compensate for another’s difficulties. The 
schedule slack was tracked both as an absolute quantity and as a relative slack-to-go based on where we 
were in the project lifecycle. 


Technically strong but numerically small project offices lower cost by reducing the cost of the contractor 
personnel reguired to service the Government side. 


The PM-led (as opposed to Pi-led) LCROSS project office was about as small as can be: 

♦ One project manager (PM) and one deputy project manager (dPM) to help ease the workload and 
provide backup. Both of these individuals were engineers, which made technical decision-making 
with the SEs smoother. 

♦ One business manager, but with an analyst in the first half of the project lifecycle. 

♦ One project control lead who handled configuration and data management, including servicing the 
online project archival site, and supporting and organizing major project reviews. 

♦ One contracting officer (CO) (part-time), who supported the acquisitions, while the dPM acted as 
contracting officer’s technical representative (COTR). The PM was the backup COTR. 

♦ One Administrative Assistant (AA) (part time) who would take care of the housekeeping and review 
support activities of the team. 

♦ One Master Scheduler who would continually update 
the schedule from all the content from the CAMs and 
NG as well as status progress to the PM. 

♦ One project systems engineer (PSE) with a secondary 
SE, who was technically quite strong in spacecraft 
systems. This was too lean in retrospect, but was the 
ultimate in technical consolidation and efficiency. 

♦ One risk manager (part-time), who maintained the project risk database and facilitated the risk 
management boards, with the PM being the Chair. 


The entire LCROSS project 
team was formulated in a 
matrixed fashion from 
organizations within ARC. 
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♦ One safety and mission assurance (S&MA) lead (part-time), who moved around to where the 
various activities were occurring and advised the SE and PM of areas of note or concern. 

This small project office handled all the activities of the mission, including the management of NG and 
was an amazingly efficient team, relying on the individual prowess of the team members and not on an 
organizational army consuming considerable resources. 


Emphasize and strengthen the autonomy of the project office. We are in an era when the project side of the 
management matrix is grossly subservient to the institutional side. The odds of conducting a successful 
project are greatly enhanced when the project office has command of its own legal, accounting, contracting, 
SR&QA, etc. functions. These should be matrixed to the project from the institutional base but, once done, 
should report to the given project for its duration. We in NASA absolutely do not do this; we reguire the project 
offices to utilize the services of centralized institutional disciplines. There is no absolutely no incentive for the 
institution to respond to the needs of the project in a timely, aggressive, efficient manner. 


The entire LCROSS project team was fonnulated in a matrixed fashion from organizations within NASA- 
ARC. The PM, project systems engineer, chief scientist, business manager, contracting, operations team, 
and the payload development team were all matrixed to the project. Most importantly, these individuals first 
answered to the PM in their role on the project. While the team members reported to their home line 
organization for issues of timecard, training, performance appraisals and the like, they were LCROSS team 
members and were managed by the project; their performance evaluations were largely from evaluations of 
the PM or CAMs on the team, since that who was determining their work function. The executive level at 
NAS A- ARC gave the PM extraordinary informal authority by way of direct communications with the top. 
If I felt that I was not getting support from an institution within the Center, or that one of my matrixed staff 
was being hampered in some way by the line institution, I had the means to bring it to the top to get 
remedied. This authority comes from the premise that the Agency and lead Agency Center needs the project 
to be successful and in order for the PM to assure the success, the staff need to be first answering to the 
project. While the line organization and the 


project manager will generally be in mutual 
alignment, there are times where this is not the 
case. Line organizational problems can surface 
which could lead to staff availability issues or 
reporting constraints which could be at odds 
with the needs of the project. The project team 
must be organized as an integrated product 
team with common purpose, unified in 
fulfilling that purpose. 


We could steer the details of the 
contractor's implementation, but then 
we'd be driving the contractor away 
from their natural, proven capabilities, 
away from their standard processes, 
driving down efficiency.. .all with the 
cash machining running. 
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Consider delegating more end-item responsibility to the contractor, with the Government in the role of data 
recipient or of proof-testing the final hardware to certify acceptance vis a vis the original reguirements. 


As noted previously, the LCROSS project created a hierarchy of requirements and then delegated the 
flow-down of those project Level-3 requirements to the providers. We could steer the details of the 
contractor’s implementation, but then we’d be driving the contractor away from their natural, proven 
capabilities, away from their standard processes, driving down efficiency... all with the cash machining 
running. If the Government starts directing the contractor on implementation details, it is steering the 
contractor away from the very experience the government wishes to exploit. The Government S&MA 
lead should have excellent insight into activities of the contractor, so as to get insight into the contractor’s 
quality of product and ability to meet the government-stipulated requirements. The government S&MA 
lead should not direct the contractor to conduct activities in a certain manner; the contractor is a 
profitable (presumably) entity who knows how to do this work, so allow them to do so. If the quality is 
slipping or there are indications of external drivers slighting the needs of the project, then the S&MA 
lead reports to the PM items of potential concern. 



Consider sole-source procurements where we know in good detail what we want, and can justify unigue cost, 
schedule, or technical advantage with a particular vendor. 



LCROSS used this approach with several of the payload instrument providers. To open up a competition 
on an instrument which existed in a catalog would have grown cost and introduced technical, cost and 
schedule risk. Remember LCROSS employed capabilities-driven approaches, which means we tried 
everywhere possible to make use of products that already existed in order to fit our tight cost and 
schedule demands. Commercially-available hardware was perfect, where appropriate, because it was 
already cost-competitive by its commercial position in the market, and was very mature since thousands 
of the products had previously been built and sold by the vendor. In effect, NASA was coming-in after 
years of other customers matured and improved the product. As noted earlier, we established excellent 
working relationships with these providers since we stressed their products enabling them to later 
improve them, all while acting in the best interests of the Government in doing so. 


The LCROSS mission approaches were very closely 
aligned with the values and processes of the NASA 
Administrator at the time of LCROSS selection; I 
find it amusing that to some the LCROSS project 
was unusual... perhaps an indication of needed 
change. I also find it amusing that while this email 
was sent out long before LCROSS ever existed, I hadn’t read it since 1993 when it was first received and 
with a retrospective read now, we rediscovered the same truths. 


The LCROSS mission approaches 
were very closely aligned with the 
values of the NASA Administrator 
at the time of LCROSS selection. 
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The LCROSS alignment on these approaches are an affirmation that we struck a good balance between 
the needs of the Agency and the needs of the industrial partner. NG was very pleased with the approach 
taken on LCROSS because they were a genuine voting member of the team and we respected what they 
brought to the table. The NASA project carried ultimate authority and responsibility, but we listened to 
NG’s expertise because they knew how to build a spacecraft. 

As a closing calibration, shortly after impact, I received a note from the NASA Associate Administrator 
who selected LCROSS during the competition in 2006 - Scott “Doc” Horowitz: 


2009 - 10-11 

Dan, 

I am sorry that I could not attend the festivities at Ames last week. Even though I could not see the 
impact, I watched from my back porch as LCROSS completed its mission. 

You and your team can take pride in accomplishing such an incredible mission is such short time and 
such a constrained budget. WELL DONE! 

While the science that will result from this mission will be incredible, the most important result will be that 
you and your team have proven that NASA can accomplish great things using this construct. 

Please pass on my congratulations to the entire team. I use your project as an example of the great 
things this nation can do when we put our minds to it and leadership makes a commitment then hands it 
over to the smart folks and gets out of the way! 

I know you have a lot to do, and I eagerly await to see what LCROSS has “unmooned”! 

Take care, 

“Doc” 


As an Agency we need to recognize 
that capability-driven missions are 
inherently better suited to live within 
their means and live side-by-side 
with the big flagship missions. 


Part of the NASA portfolio naturally costs a lot 
of money because it is doing inherently complex 
activities. However, another part of the portfolio, 
has investments that the Agency has already 
made in those expensive missions can get some 
additional return. These are the missions where 
good science can be accomplished leveraging 
what exists; where the project is encouraged to fit the science and missions within means. LCROSS is an 
excellent example of that type of design-to-cost mission. As an Agency we need to recognize that 
capability-driven missions are inherently better suited to live within their means and live side-by-side with 
the big flagship missions. The LCROSS mission may serve as a model for managing small projects. As for 
the NASA Administrator’s view of how to efficiently and cost-effectively manage NASA’s work, I found 
that many of the approaches employed by LCROSS were in very good alignment with the Administrator’s 
vision and intentions, perhaps revealing that Class D approaches are not necessarily risky, but prudent if 
well- aligned with the context of the project. . .of any project. 
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End Notes 

1. Michael D Griffin, All Hands Email, 12 Feb 1993,: Subj: “Costs of Doing Business.” 
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In 2008 the History Channel created a fascinating special called “Life After People,” in which they 
considered what would happen over time if humankind were simply to disappear. Which artifacts of 
human existence would decay immediately and which would last for some time? Some examples: 

♦ 10 days after people 

• Food would begin to rot in grocery stores and in refrigerators. Meltwater from freezers or food 
on counter tops could provide temporary sustenance for pets. 

♦ 6 months after people 

• Smaller forms of wildlife not normally seen in civilization, like coyotes and bobcats, would 
begin to inhabit suburban areas. 

♦ 1 year after people 

• The Hoover Dam would stop generating power as mussels clog coolant pipes. 

♦ 25 years after people 

• Sea water would flood into cities such as London and Amsterdam, which are currently kept dry 
by human engineered projects. 

♦ 40 years after people 

• Many wooden frame houses would have burnt down, rotted, or have been largely consumed by 
tennites. 

♦ 75 years after people 

• Many of the roughly 600 million automobiles on earth would be reduced to barely recognizable 
metal. 

♦ 100 years after people 

• Large bridges such as the Golden Gate Bridge and the Brooklyn Bridge would collapse due to 
corrosion of support cables. 

♦ 200 years after people 

• Large structures such as the Empire State Building, Sears Tower, Space Needle, and Eiffel Tower 
would collapse due to corrosion, invasive plant life, and ground water destabilizing their 
foundations. 

♦ 500 years after people 

• Items made with modern concrete would give way as the steel rebar reinforcing them expands to 
three times its original size as it rusts. 
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♦ 1,000 years after people 

• Most modem cities would be destroyed or covered in flora, with collapsed skyscrapers 
becoming new mounds and hills. There would be little evidence that a human civilization had 
existed on earth. 

♦ 10,000 years after people 

• The Hoover Dam, one of the last remnants of advanced civilization, would fail due to erosion of 
its concrete and the cumulative effects of seismic activity. 

• The Pyramids at Giza would remain, but would be mostly buried by Sahara Desert sands. 
Portions of the Great Wall of China might also remain intact. The faces at Mount Rushmore 
might also survive and remain recognizable for hundreds of thousands of years. 

Long after the LCROSS-rewritten textbooks on the nature of the moon have vanished, even after tens of 
thousands of years passing since humans, up there on the moon, crumpled in the bottom of a 
permanently-shadowed crater called Cabeus, will remain the remnants of a spacecraft that revealed the 
secret of water on the moon. And somewhere in the downy fluff of the regolith on that crater floor, lie 
nameplates carried by that LCROSS mission, containing the names of the many LCROSS team members 
and their families, preserved forever. . . 



Figure 23-1 : Two of eight engraved avionics panels engraved with 

the names of team members, stakeholders, and their families. 
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Chapter 24 — In the End... 


Too often I see behaviors within organizations, individuals, and management text books which claim 
there is a pathway to success - “Just do the following things. . My goal in writing this set of stories on 
the LCROSS Project was to illustrate that you can have templates and procedures, but they should only 
be a supportive framework from which you deviate. There is no checklist of tasks to assure success. You 
must thi nk and evaluate... and be brave. Hopefully through the stories of LCROSS I also conveyed the 
reasoning behind both what we did, what we chose not to do, and where we changed our mind. The key 
is the reasoning and leadership. You might’ve chosen a different path in any of these circumstances and 
could have also been successful. . . . It’s all about the context and your job as PM is to navigate that in the 
presence of many, many interested parties, with your own team being the most important of them all. 

The LCROSS project was not about success at all costs. Our success comes from managing a balanced 
approached which considered all forms of success (all three buckets: Cost, Schedule, Requirements). My 
goal in writing this book was to present both the techniques and the thinking that went into this 
interesting NASA mission, against an increasingly common backdrop - trying to do more with less. 

The LCROSS story is about the details of what the LCROSS team did, but more importantly, it is about 
the way we approached moving a concept through design, build, test and operations. These approaches 
include how we engaged with the stakeholders, the public and the media in garnering trust, while 
learning along the way. We learned that a very important part of the story is about how this particular 
project fits within the bigger picture - the broader portfolio. 

The LCROSS experience, including its techniques 
and approaches, is interesting and maybe even 
helpful, but our executing framework, our risk 
tolerant Class D classification, was as much about 
the operational mode of the LCROSS project as it 
was about the whole portfolio and how it should be 
planned and executed. “Class D” risk-tolerant approaches do not get you something for free; you are 
frankly accepting a greater level of risk. The benefit of executing within this programmatic framework 
comes from designing a portfolio (and messaging to the stakeholders) that this increase in risk position is 
factored into the portfolio’s planning; you can come out ahead, even if some part of the portfolio is lost. 
Class D projects should not be looked at one project at a time. Whether LCROSS failed or succeeded, in 
the end it was about accepting levels of risk to get the right things done for less money. 

This is tough stuff to sell. If LCROSS had failed, many in the community would have said, “We told you 
so.” “We already tried this and learned this lesson.” For this reason it was good that LCROSS succeeded, 
because some momentum needs to be built around these approaches. In my time since LCROSS, I am 
increasingly hearing people invoke terms for future NASA work which include terms like “Design to 
Cost and Capabilities Driven.” This is great but it is 

also clear that the Agency culture hasn’t really bought IX ROSS was really about a new, 
into what this means... yet. I cannot stress enough the balanced approach tO SUCCeSS. 
importance of the stakeholders, both inside and 


Whether LCROSS failed or 
succeeded, in the end it was about 
accepting levels of risk to get the 
right things done for less money. 


212 


Chapter 24 — In the End... 



Figure 24-1: LCROSS Mission Poster. 
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Figure 24-2: Never-published print advertisement, provided courtesy of Northrop Grumman. 
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outside the Agency, being coached on the philosophy of this approach. I think the LCROSS project, the 
program office, and mission directorate offices were only partially successful at conveying this message, 
since few people understood what LCROSS was really accomplishing. Yes, it was about the water... but 
LCROSS was really about a new, balanced approach to success. 

Meanwhile, along the way to LCROSS demonstrating this logic, a really amazing thing happened. This 
scrappy team immersed itself deeply in the goal. There was something to this team’s behavior, enabling it 
to accomplish great things. This goes back to earlier chapters wherein I talked about how everyone, 
despite their differing individual and organizational needs, played together with their eye on the same 
prize. We accomplished something very important - something great. 


LCROSS was one of those very 
special projects wherein the 
many diverse team members 
worked so well together that 
this was a prize in itself. 


At a personal level, I found another prize in being able to 
work on the LCROSS project. Many recognized this, but 
perhaps it was best captured during a ceremony of the 
changing of the guard of the NG PMs. It featured the 
regular speeches, but at this ceremony a moving message 
was delivered by one of the NG vice presidents, Tom 
Romesser. He shared that he had been a member of many 
teams over his long career and was proud of what he had brought to each one. He noted that some of the 
projects were very successful and others were not, but very few were what he would call special. He said a 
special project is one in which the team so gels together with such common purpose, that lines between 
them almost disappear. Yes, there are contracts to deal with and other mechanistic things that must be 
addressed, but they are almost effortless because the bigger goal is so clear in everyone’s mind, that 
mutual respect and even deep, lifelong friendships are born. He noted that from his view, LCROSS was 
one of those very special projects wherein the many diverse team members worked so well together that 
this was a prize in itself. “Treasure this experience,” he told us, “because gems like these come around 
very infrequently.” Tom’s words stuck with me through the whole mission and continue to resonate with 
me. He shared these comments with such sincerity and wisdom that it was impossible to ignore. 


The LCROSS story did end well. Our Centaur impact targeting accuracy was within an impressive -100 
m of our intended location (our goal was 3,500 m and our requirement was 10,000 m), and the team 
discovered an unprecedented amount of water in the crater Cabeus. This small, risk-tolerant mission that 
wasn’t likely to succeed rewrote the science books. 


Over and Out. 
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So, remember that early team picture at the beginning of this book, taken when the NASA-ARC team 
had begun the mission? I promised I’d show the final team picture, running the risk of revealing just how 
badly this project wore us out and changed the makeup of the team. This picture wasn’t taken on the last 
day but was taken in the autumn of the project. There are indeed some different people, but very few 
changed. We do look a little different, but that’s a happy group, honored to have been able to work on 
such an opportunity as LCROSS. Similar pictures exist with the broader team including NG, other NASA 
Center teammates, and the stakeholders who were all part of making LCROSS successful. Beyond the 
formal team are the families of the team members who yielded untold hours to their family members 
working LCROSS. On behalf of the team, tha nk you for your patience and pride in what we were 
attempting to accomplish. 



Figure 24-3: LCROSS NASA-ARC final team picture, 2009. Last row (L to R): Eric Due, John 
McCormick, Tony Lindsey, Brian Day, Mark Shirley, Leonid Osetinsky, Darin Foreman; Third 
row (L to R): Rich Aros, Ken Galal, Khanh Trinh, Steve Ord, Scott Sawyer, Brian Johnson, 
John Schreiner, Rusty Hunt; Second row (L to R): Paul Tompkins, Rosatina Chan, Arlene 
Schwartz, Mike Schobey, Kenny Vassigh; First row (L to R): Kim Ennico-Smith, Leonard 
Hee, Bob Barber, Daniel Andrews, John Marmie, Gil Kojima, Tony Colaprete. 
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Appendix A — Tips and Tricks of the LCROSS Project 


Throughout the LCROSS mission, we discovered many useful approaches, lessons-learned, and Tips 
and Tricks. This whole book can be thought of as a collection of tips and tricks, but for convenience, I’ve 
listed many of them here by topic: 


1 ) 



2 ) 



3 ) 



4 ) 



5 ) 



6 ) 



Project Management 

The PM provides example. The PM establishes the tone for the team. The PM establishes 
acceptable behavior, assures team health, serves as problem solver... is the glue. He is also 
the person who will take out the trash if needed. Sometimes the PM is out in front of the 
group and sometimes he is in the back watching others do their thing, but he is leading. 

The PM as orchestra conductor. The PM performs the role of conductor in an orchestra. 
He plays no instrument, so he contributes no music; but he keeps everyone moving in the 
same direction and at the same pace so that what the team “creates” is efficient, accurate, 
and even beautiful. Sometimes the musicians are not even looking at the conductor, but he 
is there to see where everyone is when needed, and he relies on the capabilities and talents 
of all the musicians for this effort to move forward. He is uniquely positioned to hear all the 
products both of individuals and of the team. 

Give yourself time to think. Every project manager is wired differently. I need time every 
once in a while to synergize - time to sit back and reflect on the direction we are heading to 
see the way forward. While we are in the midst of working current issues, what is out in 
front of us? What will stakeholders expect to see next? Are there topics I should pre-work 
now to have them in place later? 

Change is good. Every staff change was painful - both for the person I was changing and 
for me, but in the end it was the right move. The team was optimized in an honest and 
respectful fashion. When you think about it, the odds are small that the team you assemble 
at the beginning of the project will be the perfect team for the job. People have not yet 
become stressed and personalities have not fully emerged. Expect to make changes to 
optimize the team. 

Getting the right people is the whole game. Having all the necessary procedures in place, 
but with the wrong people, is a recipe for failure; however, the right people can make a 
mission successful, despite adverse policies and procedures. On lean, Class D missions, the 
bullpen depth is shallow, and so the right people make all the difference. 

Do not let problems age. Problems will fester, and you will lose the confidence of the 
team and your stakeholders if you do not acknowledge problems and navigate a way 
forward. Problems happen; what matters is how you address them. 
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7) Project trinkets are currency. The LCROSS “got 
water?” lapel pins turned everybody into kids, 
wanting more of them to hand out. This also got 
buy-in from the broader team, like vendors, who felt 
a little extra pride and ownership in the mission. 

8) Visit all members of the team. Having the PM 
take the time to come and see team members work 
reinforces a sense of relevance. Listen to what they 
have to say... see what they are doing and 
understand their problems; if appropriate, help 
solve their problems. This includes extended team 
members and vendors if possible. Behave like 
you’d want your boss to behave. 

9) Celebrate successes. Time moves quickly and the opportunities for “atta-boys” fade quickly, 
so jump on celebrating accomplishments. 

10) Be truthful as it shall set you free. The old adage is true. That being said, be measured and 
smart about what you say, when you say it, and to whom. 

11) Shared kudos is a force multiplier. Sharing gratitude and congratulations increases the 
team’s sense of unity and builds relationships and mutual respect. 

12) Good-cop / bad-cop works. You may not like this, but it does work in a combative situation. 
I never consciously employed this tactic, but such situations seem to occur naturally. When a 
person barks, it opens up the opportunity for a more reasoned, sane voice to surface and make 
things whole again. . .very interesting. 

13) Projects need a cheerleader. It is important to recognize the sense of team and its identity. 
The cheerleader could be the PM or someone else. The key is that you need someone to 
keep his or her finger on the pulse of the project. 

14) Find common purpose. The most effective way to motivate team members is to be in good 
alignment with their self-motivations. This is true both on a personal level and on a team 
level. A team of individuals and a team of teams will have a much easier time if their purpose 
is in alignment with their individual or corporate needs. Such organizations as NASA-ARC, 
NASA-LPRP, NASA-HQ, and NG would all benefit from a successful LCROSS project. 
This alignment helps assure that when times got tough, they would see it through. 

15) Steering versus reacting. It is extraordinary how much information flows in the daily 
routine of a project. It is very easy to get lost in the numerous emails, reports, calls for 
infonnation, etc. Just managing these matters can fill your every day, leaving no time to 
actually lead the team. Try to assign as much as you can to others, including your deputy, 
so that you retain the ability to lead and steer the team. Leadership requires you to be at the 
helm some portion of your time to keep the project moving in the right direction. 
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16 ) 

* 

17 ) 

* 


18 ) 

* 

19 ) 

* 

20 ) 

4 


Frankness and openness. Candid discussions and even arguments are fine as long as they 
move the dialogue forward. The moment a discussion turns into ego-stroking or theatre, 
the discussion is no longer “moving forward,” and intervention is required. 

One step at a time. Concentrate on taking the hill in front of you, not about the problems 
of taking the next hill. While you always need to be looking forward, you must not be 
stymied into inaction by not being able to see all the plays through the end of the game. 
Move out armed with the best information you have and react to changes as you go. You 
cannot wait until everything is in place. Make your best judgment with what you know and 
be prepared to change course if the information changes or your decision proves to be 
wrong. You must be “OK” with 70-80% confidence level in a decision. Tight schedules 
cannot afford extended trade studies, analyses, or tests to achieve a 95+% confidence level. 
The sooner you make the decision and move out, the sooner you will know if it was the 
correct decision, and the sooner you can correct or modify as needed. The simpler the 
design, the easier this decision process tends to be, due to fewer options being available. 

Differentiate yourself as a team. Look and behave differently than others before you. 
LCROSS was already different because we were cast with the elusive “Class D” 
designation. Because this designation was poorly defined in the Agency, people expected 
us to look and feel different and we wanted the freedom to actually be different - a 
necessary condition to accomplish what we had to, in such a short time. 

Establish relationships. Establish relationships at every level you can: peer-to-peer, 
subordinate-to-superior, and superior-to-subordinate. Also establish relationships among 
organizations and levels. The purpose is to be sure that you can resolve conflicting 
information, or if one link becomes stressed or even broken, to have other ways to address 
what is needed. 

Watch for signs of stress on the team. On LCROSS, with an inherently difficult schedule 
and a small team, people will be pushed to their limits. A few burned out, or flipped out, 
but most excelled. One person lost her way in an email to the whole team, telling them to 
“Stop! !” as she couldn’t keep up with the pace of things. Another person simply became 
resigned to not knowing how he was going to meet his requirements with the resources he 
had, and simply started asking me to tell him what he should work on next (not 
sustainable). Finally, one person on the team was generating tremendous products, was an 
absolute star and changed (for the better) the way stakeholders viewed NAS A- ARC, yet it 
took a very real physical and emotional toll on him. These were all difficult situations; this 
last one being especially hard because this individual was consuming himself pursuing 
perfection. You could see the physical manifestation of it and I needed to take 
action. . .very difficult. The bottom line is you need high-performance output from the 
team, and destructive behavior is not only bad for the individual, but for the team. Watch 
for stress throughout the project lifecycle, as different people have different capabilities 
and different conditions under which their stress peaks. 
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21 ) 



22 ) 



23 ) 



24 ) 



Watch for you and your team running out of GAS (“Give-a-$hi+”). You never want 
your team so strained or so personally compromised that they lose the will to care. . .where 
they say “to hell with it,” and don’t keep up the level of personal commitment required for 
small-team missions like LCROSS. While I did not see this outright with the LCROSS 
team, I did at times feel it was close. It was at its worst when the team was pushing hard 
resolving an issue and some in the stakeholder community were not being supportive. Prior 
to LCROSS launch, I was tasked to work on a proposal for a period of a few weeks, 
concurrent with the LCROSS activities, and I hit the wall. Very long hours and distraction 
from the ongoing mission in which I had worked so hard. . . I know what running out of 
“GAS” feels like - it is not a good experience. 

Project team versus institution. LCROSS team members were staffed via organizational 
matrixing - that is, the people who constituted the team (whether at NASA or NG) were 
from various institutional organizations, matrixed to the team for their daily 
responsibilities. This staffing included the contracting officer (CO) and business manager 
(BM), two roles where it is especially important that project needs take priority over 
institutional needs. LCROSS and NASA-ARC behaved very well in this regard. The 
LCROSS business manager had her fingers on the life-blood of the project (the money), 
and the PM needed to be able to strategize very candidly with the BM in the best interests 
of the project. The same was true with the CO regarding acquisition strategies. There are 
times when institutional needs and project needs may not be in alignment. The top priority 
must be given to the project, whose success will be in the best interest of the institution. 

Some orders come from on high. You will, on occasion, receive explicit direction from 
some of the stakeholders of your project. You will not like it. Understand that the PM has 
the responsibility, even if not always the authority, which can be frustrating. Center- or 
program-level interests may drive decisions or influences that are not necessarily in good 
alignment with project needs or what you’d choose to do. 

Choose your battles carefully. Logic might not be the currency that will allow you to 
prevail, and the cost to both you and the project might be greater than the value of winning. 
It is not a perfect world. 


25 ) 



Staff makeup. The LCROSS project had an effective blend of staff, which played to the 
strengths of everyone. Some were energetic and motivated engineers looking to get 
spaceflight experience, and others were very experienced engineers looking to do things 
differently. This created a great environment for learning, and given the relatively short 
project and mission duration, the team could see the whole process from soup to nuts. 
Young engineers learned the processes for building a spacecraft, while the experienced 
engineers reconsidered well-established rules for building spacecraft that may not have 
offered value to this particular mission. Many innovative design solutions and clever 
work-arounds to problems were grown in this healthy environment. As Tony Spear 
exhorted, “Form and motivate an excellent team, a mix of experience and bright, energetic 
youth bringing enthusiasm and new methods .” 1 
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26) Collocate teams as much as possible. Communication and decision-making are much 
faster when the critical elements of a team are collocated; this is essential to executing a 
rapid development program. 


27) 



Note to the new project manager. First-time flight project management is much like a 
racquetball game I once played with my Uncle Ray some years ago. He was an older, 
seasoned, experienced player, and I was a fit, athletic, younger, less-experienced player. 
Either one can win the game, but my seasoned uncle had me running all over the court. I 
was young and fit, so I could match anything he could volley to me, but I was doing many 
times more work than he was; he was hardly moving the whole time, just placing his shots, 
running me all over the place. He knew all the right shots, and so didn’t have to be in top 
physical form; I was in top form and needed that to make up for not having the same level 
of knowledge or skill. This was reminiscent of how it felt managing my first large flight 
project. My uncle did handily beat me, by the way. A first-time flight project PM does not 
have a lot of direct experience on which to draw. ..but he does have energy, which he can 
put into getting the needed information, consulting with others and in doing so, get 
smarter, establish confidence, and get the job done. He may expend a whole lot more work 
than a more seasoned PM, but brute force effort is required to become more 
knowledgeable. An executive at NASA-ARC once told me he’d take a new guy who is 
extremely motivated over an experienced guy any day of the week. As he put it, “ The new 
guy may make more judgment mistakes, hut is much closer to the inner workings of the 
troops and will be more responsive and innovative. ” 


Systems Engineering 


i) 



The PSE/LSE is the most important person on the team. I say this from the vantage point 
of project manager. I worked very hard. Most everyone on the team worked very hard - 
that’s expected; but the project systems engineer (PSE), sometimes called the lead systems 
engineer (LSE), has the enonnous burden of watching over all the technical development, 
making sure the pieces play together, and that the interfaces between elements work. He has 
to speak for all matters technical, both down-and-in and up-and-out through the technical 
authority (TA). The PSE has to juggle it all. 


2 ) 



Define requirements thoughtfully. Once requirements are locked in they could bite you if 
the team didn’t do its homework. It’s easy and tempting to conclude requirements writing 
and analysis early, since the impact of doing a poor job is not felt until later. But it will 
come back to haunt you. During requirements fonnulation, think carefully about how you 
will verify what you are assigning as a requirement. ... Is it feasible (or even possible)? Will 
verification be expensive because it requires expensive equipment? Look forward and 
check for sensibility. Invest time up front, as it is a lot less painful than realizing you cannot 
effectively verify a requirement later. 
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3) Manage/stabilize your requirements. From the very beginning, LCROSS tried to keep 
very tight control over its requirements, both upward and downward. We knew that 
changing requirements incurs cost and schedule risk. Keeping requirements stable is easier 
to do down-and-in (within the project), as those requirements are under your control. The 
up-and-out requirements are harder, since they are under the control of the stakeholders. 
Spacecraft and payload requirements were very stable on LCROSS; we had very few 
modifications or additions to requirements after System Requirements Review. This 
allowed the team to focus on the existing development without the distraction of changing 
requirements. We did experience stakeholder requirements change early in the project 
when the program office ownership changed from NASA -ARC to NASA-MSFC. Prior to 
this move LCROSS had only Level- 1 requirements. After the changeover we had a new set 
of Level-2 requirements, dropping the project requirements to Level-3. 1 contested this 
change since it violated the well-understood “don 7 mess with the requirements rule, ” but it 
occurred anyhow. We did keep records of the hours we spent addressing this change, but 
even though it added up to a few hundred hours, it didn’t amount to enough money to 
discourage the behavior. This was unfortunate because the real impact to the team 
momentum couldn’t be measured in hours anyhow. LCROSS was more successful in 
managing the cost-uppers associated with launch slip, which was beyond our control. I 
noted to the program office that changing the LCROSS launch date was effectively a 
requirements change to the project and necessitated a change in our budgetary profile. The 
program office agreed and funded the standing-army costs associated with keeping the 
team together. This is an important part of managing the requirements. . .if the stakeholders 
change the requirements levied on your project, then it is your responsibility to assess any 
cost, schedule, or risk associated with the changes and present it to them for a decision: 
how do they wish to move forward? 


4) 



Requirements verification takes time. I did not properly plan for the labor required to 
perfonn requirements verification at the back end of the project. This resulted in the small 
SE team having to work extra hard doing the certification of flight (COF) readiness 
activities, while simultaneously verifying all the project requirements. Put together a 
dedicated verification team (even a team of one, if appropriate) whose role is verifications 
management. 


5) 



Procedural requirements tailoring takes effort. Tailoring a NASA NPR can entail an 
incredible amount of effort to do the research, fonnulate the desired position, and then 
advocate for approval of the tailored result. The LCROSS team saw this in tailoring to the 
definition of Class D. Class D should require fewer products and analyses, but since specific 
details are not defined, it takes a fair amount of work to formulate what is appropriate, 
followed by winning over all the relevant stakeholders to make it official. NG even 
experienced this in their own tailoring. NG had a formal “waiver process,” but it was so 
cumbersome that they advocated for a waiver to the waiver process. As the LCROSS project 
systems engineer Bob Barber noted, “This is actually one of the burdens placed on Class D 
projects that needs to be addressed in the future. There needs to be more guidance or Agency- 
approved streamlining for Class D. Since the missions are supposed to be low-cost and have 
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6 ) 



7) 



8 ) 

* 

9) 

* 


limited resources, each individual Class D mission cannot afford the effort to tailor every 
NPR from scratch (7120.5, 7123.1, 7150.2, etc.). Most of the tailoring needs to be done up 
front by stakeholder organizations. ” Since the time of LCROSS, increased attention has been 
brought to tailoring needs and we now rely more heavily on local Center designations. 

Use your independent review board (IRB). Your IRB may be a stakeholder-driven 
requirement for your project, but you should make use of it. “You need to have the right 
attitude; audits are really just free consulting.” By definition, the IRB is staffed with expertise 
in various disciplines, and its goal is the success of your mission. If you find you could use 
some deliberation or an independent eye, call on the IRB. You might simply invite a few IRB 
members to an otherwise internal tabletop review, for example. This has the side benefit of 
establishing trusting relationships with the IRB and it disarms the “movie critic” role that 
sometimes exists with chartered review teams. 

Manage the pace and number of reviews. Negotiate early on with your stakeholders to 
minimize the number of reviews required of the project. This may be counter to their comfort 
levels, but quick tum-around projects cannot be overly encumbered with fonnal reviews, 
which consume a large amount of resources and time. The standard NASA reviews (MCR, 
SRR, PDR, CDR, etc.) serve a purpose, but on a fast-paced mission, come too close together. 
LCROSS had a major review every 2-3 months for the first year. The team was in continuous 
pre-review and post-review mode, attempting to accommodate post-review requests for 
action, and writing plans for the next review. These activities substantially decreased the time 
available for actual work. 

Do not underestimate the support required for reviews. The technical team is of course 
required to provide the technical content, but you must account for the whole framework of 
the review - the million little details for paper production, facilities logistics, remote login- 
viewing of the review, etc., not to mention presentation templates and setting planning and 
review sessions. You need to plan for some staff to support these efforts. 

Put review efforts where you get the best return on investment. The traditional gate 
reviews are big, heavy reviews involving many people who (frankly) don’t add to the value of 
the review; even if they are good reviewers, these reviews are not really conducive to a good 
technical discussion. LCROSS set up “design audits,” which were tabletop-type reviews in 
advance of the big reviews. They were more intimate, with limited attendance and the right 
technical expertise present. These audits allowed us to go deep and keep the formality of the 
developed materials to a minimum. This was very effective in accomplishing quality 
reviewing, plus it enabled the heavy gate reviews to be shorter and lighter, perhaps only 
summarizing the products from the earlier audit. The one problem with this approach was 
getting the stakeholders to understand their responsibilities in this process. Expecting all the 
content of the audit to be replayed at the gate review wastes the time saved, and even makes 
the team expend more time and effort. You also run the risk of accumulating a large number 
of actions from two reviews. This is a difficult concept to negotiate - but is worth the effort. 
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10) 



11 ) 



12 ) 



13 ) 



Watch out for movie critics at reviews. A “movie critic” is a person in a review who 
enjoys being critical without having any skin in the game. Make it clear to the reviewers that 
you are interested in any and all comments that can improve the finished product, and that 
everything will be considered. However, the project respectfully retains the final decision on 
what to implement. In short, you want to benefit from the experience of the reviewer 
community, but choose which actions to take moving forward. 

No self-inflicted wounds at reviews. Let the IRB write the Requests For Actions (RFAs). 
Internal team concerns should be vetted internally or at the design audits and not at the major 
review. This is not about stifling dialogue, but about managing unnecessary RFA’s which 
can be very time consuming to address and in some cases, require contract modifications to 
pay for additional contractor support. 

Use liaisons. While the LCROSS mission relied strongly on contractor best practices in 
order to leverage existing experience and talent, having even one NASA guy living with the 
contractor accomplished plenty, enabling direct insight into contractor activities. We used 
two liaisons at the two contractor locations and at our sister mission (LRO) project office. 
When an issue surfaced on the Government side, we would call our liaison, who had direct 
access to the contractor and could aid in implementing the change. This also worked the 
other way, as the liaison could see something at the contractor site and have a direct pipeline 
to the NASA project team for quick response. The liaison also provides insight, but it is 
important to respect the access you are getting. There is no quicker way to lose the privilege 
of residing with the contractor than by inserting yourself in the wrong place. You may be 
able to insist on having a liaison present, but you will quickly be shut out of the important 
dialogue if you don’t always act in the best interests of the team. Our LCROSS liaisons were 
excellent, not only becoming an integral part of the contractor team, but actually being relied 
upon to do work! They both aided the contractor team, and got on-the-job training on what 
would become their spaceflight hardware, invaluable during mission operations later (our 
liaisons became the two flight controllers for LCROSS). 

Think carefully about spares. From a technical point of view this is a no-brainer - of 
course you’d like ample spares of all types; but most Class D missions like LCROSS are cost 
constrained, so the sparing needs to be prudent. You should employ sparing when there is 
either unacceptably high risk, or where the item is so pivotal to the mission that not having it 
would be difficult or impossible to work around, including unacceptably long procurement 
lead times. An example of this for the LRO and LCROSS missions was our common spare 
single board computer (SBC). Because of the risk associated with the lead times in acquiring 
flight-qualified SBCs, the program office agreed at the LCROSS PDR to fund a spare flight 
SBC for potential use by either project. Later, when problems were discovered with the 
circuit board layout of our SBC, as well as some workmanship issues, the spare SBC played 
a key role in maintaining schedule for both projects. 
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14) Other players in your context. LCROSS needed a very sporty development and testing 
schedule in order to make the launch with LRO. This was understood from the beginning, 
but what was not fully appreciated was the pace of the launch vehicle (LV) development. 
The LV folks have a very rigid development schedule, which for their culture helps to 
assure their processes and ultimate success. As we learned on LCROSS, the Launch 
Services Program (LSP) and LV contractor processes were not fast enough to support 
quick-turn-around missions. LCROSS was at CDR 9 months after authority to proceed, 
and LSP was still at requirements review. This put LCROSS at great technical risk, since 
our spacecraft design was done before the LV was thermally, mechanically, and 
electrically defined. So by the time the LV design was finished, we really couldn’t do 
anything with the spacecraft, because it was so far ahead and we could not afford to wait 
for the LV to catch up. We had to proceed at risk - potentially great risk - with our set of 
LV assumptions as we knew them at the time. 


15) 



Over-engineering solutions. Do not lose focus of your overall risk position. It is natural 
after a problem is discovered to want to kill the problem, mitigating it back into the Stone 
Age. This is especially true with an on-orbit anomaly; however, maintaining the big picture 
is critical. Overdoing it is a sure way to bust your budget and your schedule, and even 
grow unintended derived risks from chasing a problem. 


16) 



Streamline project documentation. Standard project documents have purpose and value, 
but for fast, constrained, one-of-a-kind missions, they can be a substantial burden. The 
project plan is a good example. It is important for documenting agreements between the 
project and the stakeholders, yet the final product tends to just sit on a shelf. Is there a more 
efficient way to capture just what a project really needs and little more? Projects should work 
with the stakeholders to streamline required documentation to just what is needed. 


17) 



Big-dog benefits. Small missions like LCROSS benefit from having other, large missions 
within the same organization. Large space missions with their large funding help to 
establish capabilities within organizations that can be brought to bear on small missions. 
This isn’t about getting anything for free, but small missions cannot afford to establish 
large, capable testing facilities like large missions can. Once the facilities are established, 
the smaller missions can make use of them, including trained and skilled staff developed for 
the big-dog missions. Small, swift missions can usually fit in between large mission 
activities because they are of short duration, and have schedule flexibility. An example 
from NG on LCROSS was the development of a computer-aided factory floor tool that 
handles generation of procedures and a number of other handy activities for I&T. It was 
developed to support other large customers at NG, but LCROSS made use of this tool 
without having to pay to develop it. 
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18) Shipping the spacecraft. Cost-constrained missions require innovative ways to save 
money while still accomplishing the needs of the mission within acceptable risk. This 
includes shipping the spacecraft from the place of I&T to the launch site. For LCROSS, we 
decided to use ground shipping, which meant the spacecraft had to survive the journey from 
Southern California to Southern Florida. Here are some tips from the LCROSS launch 
vehicle manager: 

a) Assign a project “ride-along” during transport to act as an authority during shipment. 
The ride-along is the NASA representative should there be any issues and is the first-line 
authority on what to do and whom to contact. Believe it or not, many trucks are actually 
shot at during the journey across the southwest, and they encounter many other traffic- 
related issues along the way. The ride-along will be the first responder of the Interim 
Response Team (IRT), and help assess any problems. 

b) Be sure to work with your security organization to provide a threat assessment report of 
the planned route. Whether road construction, local labor unrest, or any number of 
unknowns along the travel route, it is good to have an estimate of the local conditions. 
Further, provide your security department with the names of drivers, driver license 
numbers, and vehicle license plates of the truck so that if something does occur, your 
guys are protected. NASA security will coordinate and dispatch with local police 
departments, highway patrol, and other federal facilities, letting them know that these 
vehicles have sensitive space flight hardware, and they can assist if needed. 

c) Establish safe havens with the shipping company for the unforeseen, along the planned 
shipping route. NASA Security can coordinate with other NASA Centers and facilities 
and their appropriate security offices along the transport route to set up these locations. 

d) State permitting requirements are each different and can be fluid. While we had done all 
our homework prior to shipping, we found that Alabama and Florida required highway 
patrol escort for the transport of the LCROSS spacecraft through their states (and 
required funding as well). Coordination can takes days to approve and become a real 
problem for the security, and sometimes even the health of the spacecraft. 



19) Engineering test units (ETUs). ETUs cost money; however, with them, you can create all 
sorts of unique work-arounds when issues arise, such as late vendor deliveries, on-orbit 
anomalies requiring investigation of root cause, etc. If the project is cost-constrained, but 
there is available schedule margin, you might judiciously build only essential ETUs. If the 
project is schedule-constrained, but reserves are available, build them\ If the project is 
schedule and cost-constrained (like LCROSS), consider ETUs very carefully. LCROSS did 
have access to a small number of ETUs, which proved their worth, but we could not afford 
to build many traditional ETUs typical of NASA missions. This was obviously a cost 
savings measure, but we also felt our risk exposure was already reduced by leveraging 
existing designs. I consider the results somewhat mixed. When we discovered some corner 
case conditions (unusual circumstances that could cause unexpected behavior from the 
system) in supposedly proven designs, we really could have used ETUs for testing. In those 
cases we either had to do testing on the spacecraft simulator, which interrupted the mission 
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operation team’s use of it and therefore affected our critical path, or work directly on the 
spacecraft, which carried its own risk. The best way to strike the right balance is to have a 
really good understanding of your risk exposure and determine where ETUs are worth the 
money - a tough call to be sure. 


20 ) 



Spacecraft vs. mission operations capabilities trading. On LCROSS, the name of the 
game was speed, both because of the challenging schedule, and because of the cost cap. The 
spacecraft was moving quickly through the design phase, with mission operations systems 
typically lagging behind. This often led to some tension when trade-offs had to be made 
between spacecraft capabilities and mission operations capabilities. In the interest of getting 
the spacecraft built swiftly, we faced a constant temptation to make the spacecraft less 
capable, thus passing requirements and needs onto the mission operations team. This trade- 
off needs to be studied carefully based on the risk position of the spacecraft development 
and that of the mission operations team. The trade-off also needs to be considered from a 
cost point of view, because one of the teams may be more costly on an aggregate daily basis 
than the other. In this case you find yourself pushing the more costly team to finish which in 
turn puts more burden on the less expensive team, thereby reducing your cost risk exposure. 


Schedule Management 


i) 



Schedule estimates and human nature. When planning the project schedule with members 
of your team, understand that people tend to meet their delivery dates. This is especially 
important if those delivery dates were padded in the first place. A padded schedule leaves 
little incentive for beating the schedule, since simply meeting the schedule equals success. 
The PM needs to push the schedule estimates of the team members commensurate with the 
risk position of the project. Schedules should not be based on near- 100% confidence in the 
estimate. The resultant reduction in schedule can serve as project-held schedule reserve, 
ideally stored at the back end of the project, where it can be applied to un kn own issues that 
surface throughout project execution. That reserve is the PM’s to manage as a resource and 
not to be squandered up-front when you don’t know what you don't know. Northrop 
Grumman saw huge benefits from this approach in the LCROSS propulsion system design 
and build; NASA- ARC saw the same in the development of the payloads, holding close to 
the original schedule even though the spacecraft “need date” for the payload panel was 
moving to the right for a host of avionics reasons. By holding to the original delivery date, 
we had the opportunity to either hold to the original plan or act on an opportunity to conduct 
additional testing that was not in the baseline plan, thereby using that generated schedule. 


2 ) 



Schedule and procurement sharing subtleties. Procurement planning is obviously vital to 
your schedule, but some subtleties relating to shared procurements are worth recognizing: 

a) Understand where your project’s equipment is in the build process queue at the 

manufacturer’s factory. The number of customers having products built before you may 
unwittingly expose you to additional schedule risk, even while potentially lowering your 
technical risk. 
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b) The lower priority mission may have to wait for the higher priority mission in the 
delivery of shared hardware. We discovered that the higher priority mission may require 
the first unit delivered, even if they were positioned later in the queue. Further, it is also 
possible that if there is some acknowledged minor defect on one unit, the higher-priority 
mission may get the pick of the litter, selecting the unit they wish to use. For example, if 
one of the units has seen a large number of test cycles, that would increase its available 
lifetime risk and it may get passed to the lower-priority mission, simply by being the 
lower priority or risk-tolerant mission. 

c) Once you receive the equipment you purchased, do you need access to vendor ground 
support equipment, such as an equipment test set to conduct your testing? Test sets 
facilitate checking out the hardware and could be essential to troubleshooting problems 
in your I&T phase, including issues with your flight software. On LCROSS we 
discovered that our SBC and star tracker vendors had only one test set available, which 
caused I&T schedule problems when other customers needed to use it first. 


Business Management 


i) 



Front-end load business management. The formulation phase of a project is heavy with 
activities associated with mission baselining, prior to confirmation review. This is arguably 
the single busiest time for the project’s business management team. The business staff size 
can be reduced to a more nominal level once the project is baselined or confirmed. 


2 ) 



Be leftward-leaning with your planning. We applied this approach many times on 
LCROSS. “Leftward-leaning” means doing things earlier (the left side in a schedule). We 
had to move through the design phase quickly in order to keep schedule and costs in check. 
The problem was that not every activity ended at the same time, and waiting around for 
others to complete before conducting a review would harm the schedule. If we were 
convinced that the lagging tasks represented little risk to the rest of the project, LCROSS 
was OK coming into a review with kn own forward actions to complete lagging tasks. We 
were candid about them with the stakeholders; we established a plan to complete the work, 
and the reviewers knew about these lagging items before the review. 


The spacecraft Systems Acceptance Review (SAR) is a case in point: we had a payload 
neutron spectrometer that was still having some work done, the flight software still had a 
vendor SpaceWire bus problem, the payload thermal analysis was not yet correlating to the 
spacecraft thermal-vacuum data, and we had some unanswered questions about the 
spacecraft thruster alignment; however, 95% of the spacecraft was done and ready to go. So 
we held the SAR and put it all in front of the stakeholders for their evaluation. It was very 
successful in that the stakeholders got to comment on the condition of the spacecraft and 
buy off on our approaches, and it showed that we were candid about our status. After this 
review, we were able to off-load a number of NG staff who were not related to the open 
items, and only maintained the staff required to execute the forward actions. We were also 
able to work the issues that surfaced at that time, instead of waiting to hear about them at a 
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3 ) 



4 ) 



5 ) 



1 ) 



2 ) 



3 ) 



later review. We later briefed the closure of all of these SAR open issues at the next review, 
the Pre-Ship Review (PSR). 

EVM is only a tool. LCROSS waived the Agency requirement to conduct Earned Value 
Management (EVM). This was not a repudiation of EVM, but an acknowledgement that its 
value and cost must be balanced against the capability and needs of the team. EVM provides 
a means to measure the performance of the team, but is not the only way to monitor 
performance. The key to EVM is to spot trends with your deliverables early - before they 
affect the project critical path. LCROSS continually monitored schedule performance of 
each WBS element down to level 4, as well as cost with each project CAM. This allowed us 
to successfully see negative trends before they were unrepairable, without the complexity of 
EVM formality. 

Watch your money’s age. Watch not only the expiration of year-end monies, but be aware 
of financial adjustments by your contractor; getting money returned from the contractor 
could cause problems if it’s assigned as previous-year money. 

Plan for internal overhead costs. When LCROSS was fonnulated, NASA- ARC had not 
fully defined the overhead burdening for the project. This means we hadn’t properly planned 
for what would effectively become a hen against our reserves. Be sure to plan for all the 
overhead costs, both organizational taxes and equipment taxes of your home organization. 

Contract Management 

Contractor negotiations philosophy. You must be very clear who is responsible for 
originating any contract changes. If the Government changes a set of requirements, then 
they own the cost and schedule impacts related to the change. Conversely, the contractor 
clearly has full ownership of a problem when they need to change a requirement due to an 
unplanned situation. Sometimes, however, a required change is not clearly owned by one 
party or another, and then you have shared responsibility. An example of this is when, by 
Murphy’s Law, something bad happens to your hardware through no fault of either party. 
LCROSS had a cost-plus-fixed-fee contract with NG, meaning that all costs are incurred 
by the Government, but we still behaved as if it were a fixed-price contract with everyone 
having ownership of their responsibilities. This maid negotiations cleaner and fair. 

Every change costs money. Every trade study or investigation costs money. Every event/ 
review costs money. Wherever possible, negotiate the cost before you direct the work, or at 
least cap it until you can define it better. You don’t want surprises. 

Be creative with the contractor statement of work (SOW). LCROSS placed an analysis 
task in the SOW, which was left inherently generic so that it could be used for a number of 
different purposes. We called this the “trade studies” task or bucket, and it was of huge value 
to the team. It was specifically to be applied to contractor efforts to investigate a problem or 
an opportunity at the direction of the Government contracting officer (CO). We put a small 
amount of money in the trade studies bucket, and then as circumstances warranted, we 
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would direct NG to study alternatives or to scope potential solutions to a problem, as they 
came up. This allowed the contractor to immediately work an issue, since it was already in 
scope within the SOW, and didn’t require a contract mod to execute. The next time a 
contract mod was executed, we reviewed the “study bucket” balance and assessed if the 
current landscape required additional work, and we adjusted the funding level accordingly. 


4 ) 



A good contracting officer (CO) is essential. On constrained missions, innovation is 
important for everyone - including the contracting officer. While all Federal acquisitions must 
adhere to the FARs (Federal Acquisition Regulations), it is essential the CO be willing to work 
within the latitude provided by the FARs. It is tempting to simply take the easy or convenient 
interpretation of the FARs, but a good CO will always work different interpretations of the 
rules and support the PM with choices like, “We can’t do it that way, but here’s a way we 
could maybe do it,”. . .No one wants to end up in jail by breaking Government contracting law, 
but the FARs do allow room to work a clever solution to your needs. 


5 ) 



Understand that in the end, the contractor’s bottom line is its bottom line. There is 
nothing wrong here; profit is why corporations exist. Never forget this principle. The key is 
to play to the necessary interest of the contractor, just as you would with any project 
stakeholder. Y ou always seek to get the most out of all the team members by playing to 
their needs. The contractor, as any team member, needs to perform, but if we understand its 
context, we can factor that into strategic thinking. Allow the contractor to do the work for 
which you hired them, and don’t impose additional constraints or processes. In the end, this 
saves the project money. A frank understanding of how the contractor is doing financially 
builds trust. 


6 ) 



Use contractor best practices. If you select a contractor based on their ability to deliver 
what you need (and isn 7 that the point, after all?), then don’t impose your rules upon them 
when they execute to your requirements. Allow that capable contractor the opportunity to 
be successful for you. They have a set of corporate best practices, which includes their own 
corporate oversight to assure their work. NASA traditionally adds a layer on top of that 
with its own requirements for execution, including testing and reporting needs. This adds 
cost and schedule risk, and maybe even technical risk if the NASA requirements drive the 
contractor off their game. LCROSS did not overlay a series of oversight requirements on 
top of the contractor’s normal activities, provided they let us see their work. It was about 
mutual trust because NASA allowed the contractor the freedom to execute business the 
way they normally do, and NASA in turn got insight into all the activities: a win-win. 
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1 ) 



2 ) 



3) 



4) 



Risk Management 

Risk management while risk tolerant. Risk-tolerant missions require strong risk 
management as they must detennine what will and will not be done. The NASA Class D 
designation means the project can accept a heightened residual risk position. The types of 
test you conduct, or how many cycles of a given test, and to what amplitude levels are 
determined by the specific needs of the project and stakeholder community. A Class D 
project will at times say, “We choose to not buy down this risk further because it costs too 
much or consumes too much schedule slack.” 

Risk during integration and test (I&T). During LCROSS payload environmental testing, 
we had a couple project team missteps, leading Center management to put some extraordinary 
controls in place on the handling of the flight hardware. While this may instinctively feel 
appropriate, decisions like this should be handled within the risk management framework of 
the project. Some LCROSS flight hardware were very inexpensive because they were 
essentially COTS equipment - some costing as little as $10K. If this hardware could readily 
be reacquired (i.e., replaced in a reasonable amount of time), it did not make sense to spend 
$100K in labor protecting it. This was a very difficult position to make politically, but 
through good times and bad, risk management is what should drive decisions. 

Balancing risk. When evolving the design and approach on a risk-tolerant mission, make 
sure you do not require more reliability from one component than from others in the same 
system. I remember one LCROSS risk management board discussion where we found 
ourselves chasing risk reduction of one element of the payloads to a much deeper level than 
other elements in the same system. We were not exercising balanced risk management which 
squanders resources. By taking a unifonn approach to risk, you can save money by choosing 
to do less, while still playing to the stated risk position of the mission. 

Risk-class warfare. Having co-manifested payloads with different risk classifications leads 
to class disparity and therefore difficulties during launch vehicle planning and integration. 
The required testing, analysis, and margins, along with acceptable residual risk, are all driven 
by each entity’s stated class. In the end, the highest-common denominator rules the day. No 
one is doing anything wrong - each party simply has different needs: 

a) LCROSS = Class D 

b) LRO = Class C/B 

c) Launch vehicle = Class A 

LCROSS ended up having to perform testing and analysis beyond that required for a Class D 
mission because LRO and LV required it to address their residual risk. This directly translated 
into elevated cost and schedule demands on the very project that was cast as a Class D to 
enable savings and efficiencies. 
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1) 



2 ) 



3 ) 



4 ) 



5 ) 



Stakeholder Management 

Engage your stakeholders. I found that without engagement, stakeholders got nervous. 
Without concrete information they started to think the worst - that we were holding out on 
them somehow. Engage with them while helping them understand that your team’s 
openness implies responsibilities by the stakeholders as well. 

Get stakeholder “skin in the game.” The best way to assure that the program office and 
HQ behave to your project’s advantage is to get them involved in a meaningful but limited 
way in the project’s success. Give them some fonn of ownership of a problem. You need to 
be careful with this, but if they can help solve a problem by virtue of status or experience, 
allow them to. Encourage the program office or HQ to say a few words about the project at 
the beginning of major reviews. 

Define minimum and full mission success. Common understanding between a project and 
its stakeholders on your requirements priorities is essential. On LCROSS, this alignment 
enabled us to make strategic decisions when required, because those options were 
effectively pre-negotiated with the stakeholders. A couple of times during the mission we 
had to consider dropping to minimum mission success. We never had to do it, but we 
considered it. One example was with our propellant anomaly, in which we lost a great deal 
of our propellant margin. Minimum mission success required us to have sufficient fuel to 
successfully navigate into the desired crater; however, several additional payload-based 
operations activities, (calibrations, instrument bake-outs, etc) served the needs of full 
mission success. If we couldn’t do them all, we were prepared to drop activities associated 
with the payload, despite what that meant for the data from LCROSS. 

Don’t let stakeholders manage your reserves^ After launch, LCROSS carried a fair 
amount of reserves allocated to the threat of an on-orbit anomaly, which would require 
extra support labor funding. In particular, the cost associated with NG was higher than the 
equivalent Government personnel and contractors, so this “threat” reserve amount seemed 
quite high to some stakeholders. Their concern was heightened by the fact that we wanted 
to carry this funding across a fiscal year change, which can be problematic for HQ financial 
metrics. In the end, we used those reserves for the propellant anomaly, thus validating our 
strategy. Suddenly, everyone was glad the money was there. It is very tempting as the 
Government year-end nears to collect any unspent monies for other needs, but the 
stakeholders need to understand that the money is allocated toward a threat, per good risk 
management, and is not available. Since the time of LCROSS the Agency has formally 
recognized this fact and has coined a new term, “Unallocated Luture Expenses” (ULE), 
which acknowledges that the reserves are intended to be consumed - they just haven’t been 
allocated yet. 

Consolidate reporting. LCROSS leveraged its reporting products in order to avert 
redundant efforts. It is not uncommon for multiple reports to be required every month for a 
project, but for small projects, this is a substantial burden and generally inefficient. Time 
spent reporting is time not spent doing actual work. LCROSS worked with the stakeholders 
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to hold a single monthly review at NASA-ARC, and then used the report from that review 
to communicate the same to the program office and HQ mission directorate. We had to 
negotiate a bit on what the report would cover, but once that was set, it was very simple. 


6 ) 



Stakeholder help during troubleshooting. The team must have sufficient time to work 
problems themselves. External “help” can be distracting to the ongoing activities. It is the 
PM and PSE’s job to watch the troubleshooting and remediation activities very carefully to 
see if they are becoming ineffective, and then introduce outside expertise as needed. This 
requires a tremendous amount of trust by the stakeholders, who will have a hard time not 
inserting themselves. 


End Notes 

1 . “NASA FBC TASK FINAL REPORT,” Spear, T„ March 2000. 
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✓ “This mission is not about maximum performance, but cost containment. " 

♦ Howard Eller, Advanced Concepts Chief Architect at Northrop Grumman (NG) Aerospace 
Systems, who understood from the beginning what this mission really represented. 

</ “It will be fun.... I will enjoy (repeat). " 

♦ Pete Klupar, Engineering Director, NAS A- ARC, assuring me that the early project start-up 
politics and workload would give way to pride and enjoyment - just keep telling yourself 
that.... 

</ " Faster, Better, Cheaper.... Pick two. " 

♦ Dan Andrews, LCROSS project manager (PM), but originated by others, remarking that the 
PM has knobs to turn when executing a project, but they are usually interconnected. 

v* “I stand by ready to support the cause of light and right, against the forces of dark and 

evil. " 

♦ Pete Klupar, Engineering Director, NASA-ARC, acknowledging that some of the things we 
were trying to accomplish on LCROSS were non-traditional and that not everyone would be 
supportive. 

</ " The boss is often wrong - just remember that he's still the boss. " 

♦ Pete Worden, Center Director, NASA-ARC. 

v* “Just because I am paranoid does not mean they're not out to get us. " 

♦ Quoted by Joseph Heller, http://www.goodreads.eom/author/quotes/3 167. Joseph_Heller. 

v* “Experience is a hard teacher because she gives the test first, the lesson afterward. " 

♦ Quoted by Pete Klupar, Engineering Director, NASA-ARC. 

“As we move closer to launch we move away from Class D and increasingly toward Class 

C-ness, Class B-ness, and Class Anus. " 

♦ (Name omitted), correctly cautioning that the pressure from stakeholders to be less risk-tolerant 
was growing as we approached launch. 

v* “Silence is acceptance. " 

♦ Quoted by Steve Noneman, LCROSS program office Mission Manager, NASA-MSFC, noting 
that we may need to push back on work we believe is unnecessary. 

✓ "LCROSS will be a smashing success. ” 

♦ Pete Klupar, Engineering Director, NASA-ARC, authoring one of the most clever LCROSS 
marketing puns, which I used frequently. 
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s' " ...standing by to engage photon torpedoes and phasers by your command. " 

♦ Pete Worden, Center Director, NASA-ARC, responding to an email I sent him warning that we 
had a vendor delivery issue and that while he could “push the big red button” and call the 
vendor, I thought we had a work-around at the project level. I requested he hold off, and give us 
the chance to work it. This was his response. 

s' "LCROSS will detect ice in LRO's impact plume. “ 

♦ NASA 2009 Budget Estimate Summary Strategic Goal SG6. This was an actual typo in the 
NASA budget summary that implied that LRO was planning to impact the moon instead of 
LCROSS. 

♦ “We have been having some difficulties of late... I guess they’ve begun to be reflected in our 

ops plan. ” 

• Craig Tooley, LRO PM, jokingly responding to the above typo in an email to me. 

s' "Inform. Entertain. Crow. " 

♦ Nancy Bingham, friend and coach, reminding me of the role of the PM when addressing the 
media. 

s' "Dan 'The Hammer' Andrews" 

♦ Craig Tooley, LRO PM, having fun with an article discussing LCROSS, titled “NASA Takes 
Aim at Moon with Double Sledgehammer. ” 

s' "Exploder- 1" 

♦ Brian Day, LCROSS educational lead, coining a new nic kn ame for LCROSS. 

s' "Dan is made of 90% Emailium. " 

♦ John Marmie, LCROSS deputy PM, referring to the number of emails he received daily from 
me. 

s' "Sometimes I feel like a wireless router. " 

♦ John Marmie, LCROSS Deputy PM, again pointing out that I can be prolific with email. 

s' "LCROSS - Making an impact on America's return to the Moon" 

♦ United Launch Alliance, quoted on a corporate poster. 

s' "I envy you, Dan; all you have to worry about is LCROSS. " 

♦ John Marmie, LCROSS Deputy PM, at a time when he was simultaneously working on 
LCROSS and a new mission proposal (an impossibly tough position to be in), while I merely 
had to work LCROSS. 
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v* "Clearly where you stand depends on where you sit... or where you want to sit in the future. " 

♦ Pete Klupar, Engineering Director, NASA-ARC, giving me some context on some difficult 
decisions I had to navigate. 

v* "I'm on the verge of starting to enjoy myself. " 

♦ Dan Andrews, LCROSS PM, in an email I sent to a team member, after I finally saw that this 
project was actually going to make it, and the team was executing on all eight cylinders. 

♦ “It ’s about time you started to savor this. ” 

• Larry Linton, Northrop Grumman, happy that I could now see the light. 

</ "LCROSS is small, sweet and sexy. " 

♦ Victoria Lriedensen, LCROSS program executive, noting how much the fascination with 
LCROSS was taking hold in the press. 

V "People can do more than you can think...so throw them into the breach!" 

♦ Todd May, LPRP Program Manager. 

V "We're MOS...we deliver!" 

♦ John Schreiner, LCROSS mission operations manager, making a tongue-in-cheek comment 
about the amazing progress the operations team was making despite continued doubt by some 
in the local stakeholder community that they could pull it off. It became a bit of a defiant 
mantra for the ops team. . .and I loved it. 

v* “You guys really got game on this project. " 

♦ Todd May, LPRP Program Manager, making a very welcome and appreciated comment about 
the LCROSS team after he came onboard as the new Lunar Precursor Robotic Program (LPRP) 
program manager. 

v* "Magical ponies won't appear and get this done in two days. " 

♦ Jose Cabanillas, LCROSS lead space vehicle systems engineer, reminding people in a telecom 
that the change we were discussing would not be trivial and to keep our sensibilities grounded. 

v* "As soon as there are buttons and patches, you are no longer Class D. “ 

♦ Pete Klupar, Engineering Director, NASA-ARC, noting that you can only really be risk tolerant 
while you are unnoticed. The moment you start to draw public attention to yourself, you can no 
longer be risk tolerant / Class D. He was only half-joking. 

v* "I agree that the MOS team is not ready to operate a spacecraft that does not work. " 

♦ Bob Barber, LCROSS project systems engineer, making a sympathetic comment to the 
LCROSS ops team during development, after someone on the spacecraft team claimed the ops 
team was not ready to operate the spacecraft. . .which was having problems of its own. 
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v' "Hey... when you're si ingin' hash, you're bound to drop some food on the floor. " 

♦ Ken Galal, LCROSS Mission Design lead, NASA ARC, poking fun at those harried days when 
the launch date from the program office kept moving and the LCROSS mission design team 
had to crunch and re-crunch scenarios and designs for getting LCROSS to an optimal crater on 
the moon. Ken and his team earned their stripes during this period and here he admitted that the 
analysis isn’t always polished. 

</ "We just fiy around and around, but you smash into the Moon... twice!" 

♦ Craig Tooley, LRO PM, noting the popular appeal LCROSS was acquiring in the media. 

V "There are lies, damned lies, and statistics. " 

♦ Mark Twain, quoted by Todd May, who was jokingly referring to my accounting of cost 
impacts with the ever-changing LRO/LCROSS launch date in the launch manifest. 

</ "It 's easy enough to be pleasant, when life hums along like a song. But the man worthwhile is 

the man who can smite when everything goes dead wrong. " 

♦ Disney’s California Adventure, Tower of Terror attraction. I saw this while waiting in line for 
the ride with my middle son. . .it struck me as apropos to the roller coaster ride that was 
LCROSS. 

</ “With all the big dogs in the manifest around us, we little dogs need to hunt in a pack. " 

♦ Craig Tooley, LRO PM, noting that in the world of moving launch dates, little missions can’t 
hope to push back on the big dogs of Shuttle or DoD missions. . .unless they stick together. 

v* “When your mission is a pathfinder, trust your team - they are the experts. " 

♦ Todd May, LPRP Program Manager. 

v* "Other than that, Mrs. Lincoln, how was the play?" 

♦ Quoted by Pete Worden, Center Director, NASA-ARC, releasing a little tension following the 
difficult LCROSS on-orbit anomaly and recovery. 

v* “Take calculated risks. This is quite different from being rash. " 

♦ George S. Patton. I learned of this statement about half-way through the LCROSS development 
and felt it perfectly spoke to how we were trying to manage the LCROSS project: reasoned risk 
management. 

v* "Luck favors the brave. “ 

♦ Pete Klupar, Engineering Director, NASA-ARC, modifying the quote, “Fortune favors the 
brave,” dating back to the third century B.C., with an eye to helping me make a bold decision. 
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v* “Work with people who make you feel uncomfortable with their prowess or competency - it's 

the best way to learn. " 

♦ G. Scott Hubbard, Director, former Center Director, NASA ARC. He shared this quote with me 
a long time before LCROSS was even a proposal, but it stuck with me throughout LCROSS 
execution. I surrounded myself with the best, hardest working, and most innovative people. 

✓ “They all become Class A in the end. " 

♦ Todd May, LPRP Program Manager, recognizing the inevitable inflation of risk intolerance as a 
mission moves closer to launch day. 

v* “I've never experienced such a level of openness on a spacecraft development. " 

♦ Jesse Leitner, NASA Goddard Space Flight Center (GSFC), acknowledging the very easy 
access he had to the key team members during LCROSS development. 

</ “I Hate Computers. " 

♦ Rusty Hunt, LCROSS Ground Data Systems Lead, often revealing an unfortunate reality of his 
work... throughout the entirety of the LCROSS project. 

v* “Gutsy move. Maverick. " 

♦ Anthony Colaprete, LCROSS Chief Scientist, in an email he sent to me following the capacitor 
decision (going back into the spacecraft to assure we had no problems). Turned out to be one of 
the finest risk management moments on LCROSS. 

* "Senior management's oversight of projects should be by commander's intent and not rudder 

control. " 

♦ Todd May, LPRP program manager, illustrating his view of how his role should mesh with 
mine. He recognized the importance of the project manager’s authority and his sole ownership 
of “rudder control.” 

V “If this is off-the-shelf, I want to see that shelf. " 

♦ Pete Klupar, Engineering Director, NAS A- ARC, noting the inevitable difficulties associated 
with using off-the-shelf parts. 

* “They have not been told this cannot be done. “ 

♦ Steve Hixson, Vice President, Advanced Concepts for Northrop Grumman Aerospace Systems 
Sector, noting that the LCROSS team was breaking all the rules of how long things are 
supposed to take. . .and were being successful with it. 

v* "The project's greatest strength is its transparency. " 

♦ Steve Noneman, LPRP Mission Manager. 
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“They're snakes - do not trust anything they tell you. " 

♦ (name omitted), Senior executive, NASA-ARC, expressing frustration over some of the 
partners early on with LCROSS. This was a very difficult time because I had to work with 
people that local center management were casting as untrustworthy. 

v* " NASA Takes Aim at Moon with Double Sledgehammer" 

♦ Space News, 2/27/08. 

</ “It's great to have hard work rewarded, but meaningful work, there's something to be 

thankful about. “ 

♦ Oliver Nguyen, friend, taking stock of the potential great significance the LCROSS mission 
could have on future exploration. 

v- “F**kin Florida." 

♦ Tom Ajluni, LRO spacecraft systems engineer, after he and I saw a moth flying around the 
Class 100K clean room immediately before LRO and LCROSS were to be encapsulated in the 
fairing for launch. One of the more surreal moments on LCROSS, as we were all donning our 
full body gowns and hair bonnets to keep everything clean. . .and a moth went flying by all this 
delicate hardware. 

</ “Adversity introduces you to yourself. " 

♦ Quoted by Linda Timucin, NASA-ARC, offering perspective following the LCROSS on-orbit 
propellant anomaly. 

v* “I'm gonna pee myself. " 

♦ (name omitted), LCROSS team member, in a text message sent to me while we were on 
console at the launch site, about to watch the culmination of our team’s hard work, head to the 
moon. Launch day at 6/18/09 at 2:25 pm. 

</ “We didn't have enough money to maintain the systems engineering team, so after the 

verification review I realized I was now the least useful person so to save money I fired 

myself. " 

♦ James Wehner, Northrop Grumman’s Lead Spacecraft Systems Engineer and Deputy Program 
Manager, taking cost-savings to heart. 

v* “Keep the speed below Warp 9 for the first 100,000 miles, ok?!" 

♦ Craig Elder, LCROSS Spacecraft PM at NG, sending the team a stress-relieving bit of humor 
shortly after LCROSS launch. 
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s' "Bam BAAM BAMMM...SIap Slap Jab...Jaaaab! POW! Love it! Wait...still standing. Hit again 
before (omitted) gets up. " 

♦ John Marmie, LCROSS deputy PM, in an email note he sent me supporting a battle I was 
having with someone in management. 

s' "Just heard from the food catering guy (our sons play on the same soccer team) that WGS 

(Wideband Global SA TCOM) has slipped launch. More on Monday. “ 

♦ Chuck Tatro, NASA Launch Services Program, revealing a new potential threat to LRO/ 
LCROSS launch date. . .that he heard from the most unexpected place. 

s' "Hire the best people for the job - not just the smartest. “ 

♦ Todd May, LPRP Program Manager. 

s' "Demolition derby astrophysicists" 

♦ KQED QUEST’S TV story, LCROSS: Rocket to the Moon. 

s' "If John 's the MOM, does that make Dan the DAD (DAD = decider and deflector)?” 

♦ Rusty Hunt, LCROSS Flight Director, having a little fun with the mission operations manager 
acronym “MOM” in an email to the team. 

s' "This is going to be fun. ..someday. " 

♦ Todd May, LPRP program Manager, in an email to me as LCROSS neared launch. 

s' "The pebble sometimes does not have any say in the progress of the landslide. " 

♦ Jose Cabanillas, LCROSS lead space vehicle systems engineer, observing that LCROSS simply 
didn’t have a place at the table in some decisions from above. 

s' "What's next, locusts in Madrid?" 

♦ Rusty Hunt, LCROSS Flight Director, when high winds in Canberra Australia threatened to 
shut down the ground stations there, and southern California wildfires were causing 
evacuations at NASA-JPL, all in the midst of the LCROSS spacecraft emergency. 

s' "The team continues to impress, providing dollars of value for dimes invested. " 

♦ Pete Klupar, Engineering Director, NAS A- ARC, in an email to management, following initial 
release of LCROSS lunar impact data. 

s' "LCROSS is a faster, cheaper and good enough spacecraft. " 

♦ Steve Hixson, Vice President, Advanced Concepts for Northrop Grumman Aerospace Systems 
Sector. 
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v* "There was a time when developing the LCROSS spacecraft and instrument payload was 
thought improbable at best...that we'd be fortunate to merely bust our budget, let alone 
successfully build a spacecraft in time. " 

♦ Dan Andrews, LCROSS PM. 

✓ "LRO/LCROSS is our first important step to knowing the moon as well as we know Mars. " 

♦ Rick Gilbrech, former Associate Administrator, NASA ESMD. 

W "Congratulations to LCROSS on your innovative approaches and tactics. “ 

♦ Michael Griffin, former NASA Administrator. 

v* "I would like to see an LCROSS-Hke science mission every year. NASA should pursue a 
common architecture/bus for doing this. " 

♦ Alan Stern, former Director of the Science Mission Directorate (SMD) at NASA-HQ, in a 
discussion with NASA-ARC Director, Pete Worden. 

</ "LCROSS is the simplest spacecraft we 've ever had to process here at Astrotech. " 

♦ Gerard Gleeson, lead planner for Astrotech Space Operations preparing the LCROSS 
spacecraft for launch. He was acknowledging our team goal of “ship and shoot,” wherein we 
did essentially everything needed prior to shipping and then sent the spacecraft to Florida to 
“shoot” (launch). 

v* "The LCROSS team has been one of the best projects I've had the pleasure to be associated 
with. " 

♦ Todd May, LPRP Program Manager. 

v* "This is some crazy $hi+, so who knows?" 

♦ Anthony Colaprete, in an email to me, after looking at some of the very interesting initial 
instrument results in the LCROSS impact data. 

✓ "PM [project management] is as much about enabling as it is controlling. “ 

♦ Todd May, LPRP Program Manager 

</ "hello dis is neer from india.., i m intrest on missions of any dangerous activities. .if u get 
me a chance i wanna go for a dangerous mission on moon :) i can do everything" 

♦ Neer, from India, in an email he sent me on Nov 12, 2009. Following the exciting coverage and 
results of the LCROSS mission, Neer (whom I do not know) was prompted to send me this 
email message expressing his desire to participate in lunar exploration. . .one of many inspiring, 
and amusing, notes I received over the course of the LCROSS project. 


243 


Appendix B — Wisecracks and Wisdom 


244 



Appendix C — Tips and Tricks of the LCROSS Project 


LCROSS Mission: 

♦ Launch: June 18, 2009 at 2:32 p.m. PDT (5:32 p.m. EDT). LCROSS was a co-manifested mission 
with the Lunar Reconnaissance Orbiter (LRO). LCROSS was managed by NASA-ARC, while LRO 
was managed by NASA-GSLC. 

♦ Launch Site: Cape Canaveral Air Lorce Station, PL, Launch Complex 41 

♦ Launch Vehicle: United Launch Alliance Atlas V (401) 

♦ Atlas Fuel: The first stage was powered by kerosene (RP-1) and liquid oxygen (LOX) and the 
Centaur upper stage was powered by liquid hydrogen (LH 2 ) and LOX. 

♦ Orbit: LCROSS had a Lunar Gravity-Assist, Lunar Return Orbit (LGALRO) around the Earth- 
Moon system at approximately 80 degrees from the ecliptic plane. 

♦ Orbital Period: Each LGALRO orbit was approximately 37 days. 

♦ Number of LGALRO: 3 

♦ Total Distance Traveled: Approximately 5,600,000 miles (9,000,000 km) 

♦ Impact: LCROSS impacted the South Pole of the Moon on Oct. 9, 2009 at approximately 4:3 1 a.m. 
PDT in a selected area of a crater called Cabeus (Changed from Cabeus A). 

LCROSS Shepherding Spacecraft: 

♦ Dimensions: The spacecraft was 79 inches (2 m) tall and 103 inches (2.6 m) in diameter. From 
“omni — Z” to “omni +Z” antennae the spacecraft was 131 inches (3.3 m) wide. 

♦ Mass: The total mass at launch was 1,964.6 pounds (891 kg), consisting of 1,290 pounds (585 kg) 
for the spacecraft and 674.6 pounds (306 kg) of hydrazine fuel. 

♦ Additional Information: At impact, the mass of the LCROSS Shepherding spacecraft was 
approximately 1,359 pounds (617 kg). 

♦ Power: Power to onboard systems was provided by a fixed 600-watt peak power solar array and a 
Li-ion battery. A star tracker assembly and 10 coarse sun sensors maintained orientation to the sun. 

♦ Targeting Accuracy: The LCROSS mission had a targeting accuracy requirement of a 12.4 mile 
(20 km) circle; however the team managed to improve that accuracy to an estimated ~83 meters 
(±66m) diameter, based on infrared and thermal camera images, compared to LRO’s LOLA (Lunar 
Orbiter Laser Altimeter) maps of the lunar coordinate frame. 

♦ Telemetry: Spacecraft communications were provided through two medium gain antennas 
operating at 1.5 Mbps (nominal), two omni-directional antennas operating at 40 Kbps (nominal), 
and a 7-watt S-band radio frequency transponder. 

♦ Data: Spacecraft data (engineering and housekeeping) and science instrument data were relayed in 
real time to the LCROSS mission and science operations teams. 
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♦ Spacecraft Provider: Northrop Grumman Aerospace Systems, Redondo Beach, CA, and Northrop 
Grumman Technical Services, Lanham, MD. 

♦ Build: The spacecraft was designed and built at Northrop Grumman’s Redondo Beach, CA, 
facilities, and the science payload was designed and built at NASA- ARC, Moffett Field, CA. 

♦ Science Payload: The science payload consisted of two near-infrared spectrometers, an ultraviolet- 
visible light spectrometer, two mid-infrared cameras, two near-infrared cameras, a visible camera, 
and a visible high-speed photometer. Data from all nine instruments in the LCROSS science payload 
were managed through a common data handling electronics unit. The LCROSS science instruments 
were largely commercially available (one spectrometer, for example, is normally used to test carpet 
fibers for recycling). The team took these instruments and ruggedized them into flight hardware in a 
fraction of the time and cost of more traditional approaches. 

The instruments underwent a rigorous suite of tests, including thermal vacuum and vibration tests, in 
order to be sure that they would survive the environments both of launch and of operations in space. 
Engineering test units (ETUs) were subjected to conditions in excess of what was expected during 
launch and in flight (proto-qualification level testing), including random vibration, shock, and 
thermal cycling. This ETU testing demonstrated that the basic design of the instruments could 
survive launch and space flight. 

After this development period, flight units for the instruments were built and tested again at 
expected flight levels (with margin) (acceptance level testing). This testing demonstrated that the 
workmanship on each of the flight units was acceptable. 

LCROSS instruments had various purposes. Visible light, near-infrared and mid-infrared cameras 
observed the shape and structure of the cloud of lunar material created by the LCROSS collisions 
with the Moon. The cameras also helped scientists image the impact crater that the Centaur rocket 
stage created. Scientists used near-infrared camera data to produce water concentration maps. Visible 
and near-infrared spectrometers obtained information to help scientists study grain properties within 
the cloud and look for clues that would indicate water, hydrocarbons and organics. A total luminance 
photometer measured the impact flash brightness and how it changed. 

♦ Mission Duration: The mission duration was approximately 113 days. 

LCROSS Centaur (Upper Stage) Rocket: 

♦ Centaur as Impactor: The LCROSS mission consisted of the Shepherding Spacecraft and the spent 
Centaur upper stage rocket that was used as a kinetic impactor, targeting a permanently shadowed 
crater near the lunar South Pole. The Centaur was the upper stage of the two-stage Atlas-V rocket, 
which brought both LRO and LCROSS to the Moon. 

♦ Dimensions: The Centaur rocket (LCROSS Impactor) was 41.6 feet tall (12.68 m) and 10 feet (3 m) 
in diameter. Attached to the LCROSS spacecraft, the stack measured 47 feet (14.3 m). 

♦ Mass: The total mass of the Centaur at impact was 4,958 pounds (2,249 kg). The Centaur had 
approximately the dimensions of a standard school bus but about % the mass. 
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LCROSS Swing-By and Cruise Phase: 

♦ Science Payload Calibrations: The science team conducted a series of activities to calibrate the 
LCROSS science instruments. The largest of the calibrations was during the Lunar swing-by five 
days after launch. The spacecraft surveyed three target craters and the limb of the Moon. An Earth- 
Moon calibration caught the double crescents of the Earth and the Moon at a distance of 323,296 
miles (520,294 km) from the Earth and 547,335 miles (880,850 km) from the Moon. 

♦ Cold-Side Bakes: The LCROSS mission carried out maneuvers that exposed the side of the 
spacecraft and Centaur opposite the solar array to sunlight. The purpose was to remove any 
volatiles, including water that had accumulated on the surface of the Centaur before launch. These 
volatiles could have complicated observations and added to Centaur targeting error by baking off 
during descent to the lunar surface, and introducing side forces on the impactor. 

♦ Separation: The Centaur and Shepherding Spacecraft separated approximately nine hours and 40 
minutes before Centaur impact, at a height of about 25,000 miles (40,000 km) above the surface of 
the Moon. 

♦ 180° Maneuver: After separation from the Centaur, the Shepherding Spacecraft performed a 
braking maneuver to create a time lag between the Centaur and Shepherding Spacecraft impacts, 
and to orient the science payload toward the Moon. 

LCROSS Impacts: 

♦ Speed: At impact, the Centaur and Shepherding Spacecraft were traveling over 1.73 miles per 
second (2.8 km/s). That’s 103 miles per minute (6200 miles an hour), more than twice the speed of a 
bullet. 

♦ Timing: The Shepherding Spacecraft was 373 miles (600 km) above the lunar surface at Centaur 
impact. It hit the lunar surface approximately four minutes later. 

♦ Angle: The vehicles impacted at about 85 degrees to the lunar surface. 

♦ Impact Crater Sizes: The Centaur impact was expected to excavate more than 350 metric tons of 
lunar material and create a crater 66 feet (20 m) in diameter to a depth of 13 feet (4 m). The 
Shepherding Spacecraft impact was expected to excavate an estimated 150 metric tons, with a crater 
46 feet (14 m) in diameter to a depth of 6 feet (2 m). 

♦ Plume Height: Most of the material in the Centaur debris plume remained at lunar altitudes below 
6.2 miles (10 km). The plume’s highest visible point was expected to be approximately 6 km above 
the surface, depending on the nature and character of the lunar regolith (soil). 

♦ One-two punch: Following four minutes behind the Centaur, the Shepherding Spacecraft flew 
toward and eventually though the debris plume created by the Centaur, collecting and relaying data 
back to LCROSS mission control at NASA-ARC before impacting the surface and creating a second 
crater and debris plume. The impacts provided a wealth of data about the mysterious permanently- 
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shadowed craters at the lunar poles. LCROSS was the first in-situ investigation of these craters to 
determine the presence or absence of water ice at the impact site. 

♦ The LCROSS team conducted numerous laboratory simulations and numerical models to create 
predictions for the two debris plumes and their resultant impact craters. However, since little is 
known about the Moon’s permanently shadowed regions, this unique mission may yield interesting 
conclusions even beyond the question of water ice. 

♦ The floors of permanently shadowed craters may not have been exposed to sunlight for perhaps 
billions of years and thus are very cold. These regions are “cold traps,” which can collect and 
preserve volatile material, possibly including water molecules. Data from LRO indicated that these 
craters could be the coldest places in the solar system. 

♦ During impact the Centaur upper stage and LCROSS spacecraft crumpled and broke apart. Most of 
their material heated up several hundred degrees during impact and stayed within about 20 meters of 
the impact crater. Any small, remaining unspent rocket fuels (primarily hydrazine) most likely 
burned up at impact. The project adhered to strict residual propellant venting requirements to 
prevent contamination or distortion of the science measurements of the impact plume. 

LCROSS Target Crater: 

♦ Crater Name: Cabeus 

♦ Crater Location: South Pole Region. 84.9° S and 35.5° W 

♦ Crater Diameter: Approximately 60.1 miles (97 km) 

♦ Crater Depth: Approximately 2.2 miles (3.5 km) 

♦ Impact Crater: Cabeus crater was an optimal location to determine if water ice exists at the South 
Pole of the Moon, given LCROSS launch date and lunar/solar geometries. It was selected using four 
criteria (in priority order): 

1) Solar illumination of the ejecta cloud (required to enable LCROSS Shepherding Spacecraft 
measurements) 

2 ) Association with high concentrations of hydrogen (thought to be sites of potential water ice) 

3 ) Observable by Earth assets (to maximize alternate plume viewing from Earth) 

4 ) Crater floor with flat smooth terrain (to maximize impact plume and excavation 
effectiveness) . 

♦ The target crater was selected using the latest data from observation assets on the ground, in Earth 
orbit and in lunar orbit, including data from the Lunar Reconnaissance Orbiter, the ISRO (Indian 
Space Research Organisation) Chandrayaan- 1 and the JAXA (Japan Aerospace Exploration 
Agency) Kaguya spacecraft. 
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♦ The target crater was changed from Cabeus A to Cabeus after a general consensus of lunar experts 
led by the LCROSS science team that Cabeus shows the greatest probability of elevated hydrogen 
concentrations at the South Pole. 

LCROSS Mission Operations Center: 

♦ Mission and science operations for the LCROSS mission were located at NASA’s Ames Research 
Center (ARC), Moffett Field, CA. A remote operations center was located at two Northrop 
Grumman facilities in Redondo Beach, CA and Lanham, MD. The Northrop Grumman team 
provided backup and additional data analysis. 

LCROSS Project: 

♦ The LCROSS mission cost $79 million. Launch manifest delays were funded by the Lunar Robotic 
Precursor Program from October 2008. 

♦ LCROSS was a NASA Class D mission that was inherently risk-tolerant. This classification allowed 
the team to meet cost, time and mass constraints in their approach to spacecraft development. In 
running a Class D mission, the LCROSS team implemented carefully considered risk management 
approaches, resulting in an efficient testing program that enabled them to understand how the 
spacecraft would operate in space. 

♦ NASA-ARC managed the LCROSS mission, conducted mission operations, and developed, 
integrated and tested the payload instruments. Northrop-Grumman designed and built the spacecraft 
and assisted NASA-ARC with launch vehicle integration at the Cape. NASA-ARC mission 
scientists conducted science mission operations in conjunction with a broad investigator community. 

♦ The spacecraft had few redundant systems (in order to keep within the cost and schedule 
constraints), but the project team kept risk in check by implementing a simple, robust design with no 
moving parts. 

♦ LCROSS was a fast-paced, low-cost mission that leveraged existing NASA systems, commercial 
off-the-shelf hardware, and Northrop Grumman spacecraft expertise. 

♦ The LCROSS spacecraft went from concept to ready-to-fly (Systems Acceptance Review) in an 
impressive 29 months. 

♦ The LCROSS payload was developed, built, tested and delivered by NASA-ARC for integration to 
the spacecraft in less than 15 months. 

♦ The LCROSS team was small, dynamic, and thought ‘out of the box’ to meet and overcome the 
many challenges posed by a low-cost, fast-paced mission. 

♦ LCROSS was a mission of opportunity; created when a change in launch vehicle for LRO resulted 
in an increase in launch capability. Exploration Systems Mission Directorate (ESMD) developed a 
basic set of guidelines and competed a new small mission to launch with LRO. NASA-ARC, in 
partnership with Northrop Grumman Space Technologies, won the competition with a proposal for a 
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low cost (< $80M), less than 1000 kg, kinetic impactor that would excavate water ice from the 
Moon, if it were present. The program office noted that if the project were to breach $79M, weigh 
more than 1000 kg, or fail to meet schedule with the launch of LRO, it would be subject to a 
termination review. 

♦ Designing an instrument package for the mission from scratch would have taken too long and would 
have cost too much. The payload, from the time the project was awarded, had to be delivered to the 
spacecraft for integration in 18 months. In addition to the tight schedule, the LCROSS science and 
payload budget was necessarily small (only about 4% of the total mission cost). Rather than drive 
the design of the instruments solely on science goals, LCROSS used existing commercially 
available instruments in combinations to achieve the necessary measurements. 

LCROSS Mission Context: 

♦ LCROSS created two small craters within the much larger Cabeus target crater and did not affect the 
Moon’s orbit. The impact craters created by LCROSS are too small to see from Earth with the naked 
eye or even binoculars. 

♦ LCROSS differed from the natural barrage of material impacting the Moon in that we decided 
exactly when and where it would hit. LCROSS targeted a specific region of scientific interest on the 
Moon and also planned many observations of the LCROSS impacts from both ground and space- 
based assets. 

♦ LCROSS ’s purpose was to confirm the presence or absence of water ice or hydrated minerals at one 
of the Moon’s poles. By “absence,” we mean that not finding H 2 0 would have been as significant as 
finding water ice. LRO and other missions have further identified sources of OH and H, but only 
LCROSS actually touched the surface of the Moon to confirm whether the source of hydrogen in the 
shadowed polar craters was water ice or captured hydrogen. The instruments on LCROSS were able 
to detect even very small amounts of water and other volatiles. Almost nothing was known about the 
form or distribution of the hydrogen within these carters. LCROSS tested theories as to how the 
hydrogen could be distributed: is it unifonn and diffuse or is it in localized, concentrated pockets? 

♦ Water will be a precious resource to lunar explorers, and lunar water could help to facilitate the 
development and sustainability of exploration missions. It could serve for various uses, including 
rocket fuel and breathable oxygen. 

♦ The LCROSS mission represents a new and innovative way of using the Centaur upper stage. 
LCROSS towed the Centaur second stage to the Moon to use it as a kinetic impactor. The Centaur 
impact threw up a plume of dirt and rocks. It was timed so that the plume would be backlit by the 
sun and therefore observable from Earth. LCROSS itself flew through the plume with near-infrared 
spectrometers that showed the presence of water and other volatiles in the plume. 

♦ Missions from Lunar Prospector to Chandrayaan-1 have detected very large amounts of hydrogen at 
the lunar poles, especially near polar craters that do not receive direct sunlight. Additionally, the 
Clementine spacecraft made radar observations that suggested deposits of water ice. Data from 
orbiting spacecraft have refined these observations but by actually touching the surface, and 
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‘kicking up some Moon dust,’ the LCROSS mission exposed water ice and lifted it into the sunlight 
where it could be studied. 

♦ The Moon is constantly bombarded with meteorites and space debris. On average, it receives three 
to four LCROSS-sized impacts monthly. Furthermore, in the past, much larger impacts formed the 
craters one can see with the unaided eye. The crater LCROSS crashed into is approximately 100 km 
across. By comparison, the LCROSS primary crater is about 20 meters across, or about 5000 times 
smaller in diameter. 

♦ Lunar Prospector (LP) orbited the Moon from 1998 to 1999. During its mission, its instruments 
revealed hydrogen concentrations suggestive of water ice in the vicinity of permanently shadowed 
craters at the Moon's poles. At the end of its mission, instead of letting Lunar Prospector randomly 
crash on the Moon, mission managers modified the mission plan to attempt to take advantage of 
targeting a specific location on the Moon, potentially enabling identification of the hydrogen 
signatures. The spacecraft was directed to hit the permanently shadowed Shoemaker crater, near the 
lunar South Pole. Scientists studied the impact but did not detect signs of any ejecta (lofted dust), so 
no measurement of water ice could be made. 

This did not mean there was no ice; Lunar Prospector was not designed to find ice. The LP crash 
was not optimized for an impact experiment. It was very small, about 160 kg, and impacted at a very 
shallow angle (< 7°). Furthermore, mission controllers were not able to accurately control where the 
LP spacecraft crashed, so there is a good deal of uncertainty as to where it actually hit. LCROSS, on 
the other hand, was specifically designed to excavate and detect possible lunar ice, both by its Earth 
(not lunar) orbit, and by the sheer size of the impact mass. 

♦ What are the differences between Lunar Prospector and LCROSS? Lunar Prospector had a mass of 
only 355 pounds (161 kg). The LCROSS Centaur impactor had more than a dozen times the mass - 
nearly 5,000 pounds (2,300 kg). Lunar Prospector impacted with a speed of 1 mile/s (1.69 km/s) 
while LCROSS hit at a significantly faster 1.73 miles/s (2.8 km/s). Lunar Prospector hit at a very 
shallow angle, only 6.3 degrees, while LCROSS came in steeply, at approximately 85 degrees, 
resulting in a larger impact and debris plume. This difference comes from the fact that LCROSS was 
in Earth orbit, while Lunar Prospector was in lunar orbit. The LCROSS Shepherding Spacecraft 
flew into the impact plume, to study it at very close range. Therefore, the LCROSS mission was far 
more efficient at excavating, revealing, and detecting lunar ice than Lunar Prospector. 

♦ There is about 12,500 square km of permanently shadowed terrain on the Moon. If the top 1 meter 
of this area were to hold 1% (by mass) water, that would be equivalent to about 4.1 x 10 11 liters of 
water! This is approximately 2% of the volume of the Great Salt Lake in Utah. The LCROSS impact 
excavated a crater approximately 20 meters in diameter, or around a hundred-millionth the total 
permanently shadowed area. It is safe to say the LCROSS impact did not have a significant effect on 
lunar water. 

♦ Lunar water could potentially be used to produce water for drinking or crop irrigation; if broken 
down into its components (oxygen and hydrogen), it could be used for generating breathable oxygen 
(0 2 ), or H 2 , which can be used to make rocket fuel. Water might also be mixed with lunar regolith to 
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make a type of cement for habitat construction. And it could be studied for potential information 
about the beginnings of the solar system. 

♦ Lunar Prospector carried a small amount of Eugene Shoemaker's ashes to the Moon as a tribute to 
his lifetime of outstanding work. Shoemaker was a planetary scientist who contributed greatly to our 
understanding of the Moon. He died in an auto accident in 1997. Since then, NASA has become 
more aware of the importance the Moon has to people of many cultures. Because of this 
appreciation of various cultures, the LCROSS Shepherding Spacecraft and the Centaur impactor did 
not carry human remains. 

♦ The LCROSS mission was dedicated to the memory of Walter Cronkite, a great advocate of NASA 
and space exploration, who passed away on July 17, 2009. 


End Notes 

1. LCROSS Point Paper, Ver 7.0” NASA Ames Research Center, October 5, 2009. 
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During the LCROSS propellant anomaly, shortly after we understood the degree of the problem for 
LCROSS, we declared “Spacecraft Emergency” with the DSN (Deep Space Network) and with the 
stakeholders. This declaration enabled our getting priority access to a lot of telemetry resources and 
bandwidth in order to get us time to save the mission. Such a declaration necessarily comes with getting 
a lot of attention as well, not just from the LCROSS stakeholders, but from the entire space-faring 
community and shortly thereafter, the mainstream media. Our declaration enabled us to utilize other 
mission’s telemetry assets in an effort get out of our emergency condition. So, the LCROSS emergency 
not only affects the LCROSS mission but also other missions who yield their assets to the ailing mission. 

Below are my email notes during the incident, presented in chronological order, just as presented to the 
LCROSS stakeholders. These notes were authored by me and the LCROSS deputy PM, John Marmie. 
Prior to LCROSS launch, I had created an email distribution list named “LCROSS Anomaly,” which was 
specifically designed for this purpose. I had hoped it would never be used, but was glad to have it when it 
was needed, in order to keep all the necessary stakeholders well-informed of our status. In the early hours 
of the anomaly, updates were fairly frequent because of fast pace of reactions and triage. As issues got 
resolved, the status updates spread out a bit. 

I show the notes here in their entirety for educational purposes. References to “Status Email sent” means 
that this was a moment where an email was sent to the stakeholders (by either me or the dPM) with the 
content that followed. 






3:50 am PDT, 8/22/09: I received call at home from MOM (mission operations manager) indicating they 
completed acquisition of signal (AOS) and found that: 

♦ Came up AOS, observed IRU [Inertial Reference Unit] and Kalman Filter faulted and disabled 
(AC DISABLE). ACS [Attitude Control System] defaulted to using star tracker for rates. Observed 
unexpectedly low propellant mass. 

♦ The IRU fault trip automatically moved into using STA [star tracker assembly] rates. Once on STA 
control we started burning a lot of propellant. It is estimated more than 100 kg was consumed - on 
the order of our entire propellant margin. 

♦ MOM [mission operations manager] declared a spacecraft emergency with DSN [Deep Space 
Network] to get additional coverage on both primary and redundant stations. They have complied; 
however, there is a geometric outage coming at 2 pm EDT, and lasting for 9 h 16 min, before 
LCROSS will come back into view. 

♦ GAVRT [Goldstone- Apple Valley Radio Telescope] didn't call because they were out of view of the 
spacecraft (couldn’t see the transponder coming on) 

♦ [Approximately] 100 kg of fuel is estimated burned; on the order of our remaining propellant margin 
I headed in to NAS A- ARC. 
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4:20 am PDT, 8/22/09: 1 arrived at MOCR @ NASA-ARC: 

♦ MOM informs me they rebooted the IRU (power-cycle) and it has come up healthy/nominal which 
has restored normal operations and propellant expenditure (very low consumption). 

♦ We are not in safe mode, nor were we ever; we are in nominal STA-based control, once the IRU 
faulted, per the design. The problem appears to be that this STA-based control was not behaving - 
consuming the propellant. 

♦ We are now focused on the task of understanding what happened and are downloading virtual 
recorder (VR) data during the time of this occurrence (while we were out of coverage); this will be 
used to understand root cause and then formulate a plan forward. 


• r««-/ 

4:45am PDT, 8/22/09: I called Pete Klupar [Director of Engineering at NASA-ARC] cell to infonn him 
of status; left voicemail message. 


4:50am PDT, 8/22/09: I called Danny Harris [LCROSS program office mission manager] cell to infonn 
him of status; left voicemail message. 


4:55am PDT, 8/22/09: I called Victoria Friedensen [LCROSS Program Executive] cell to inform her of 
status. 

♦ I explained everything, answered her questions 

♦ She infonned me she would be calling Doug Cooke [Exploration Systems Mission Directorate 
Associate Administrator] and letting Jay Jenkins [ESMD Advanced Capability Division] kn ow 


5: 15am PDT, 8/22/09: I called Dan Bufton [Deputy Director, NASA-ARC Code P] at home; he will call 
Worden [Center Director, NASA-ARC]. 


5:33am PDT, 8/22/09: Todd May [Program Manager, Lunar Precursor Robotic Program] called me after 
speaking to Danny Harris (where I left a message above). I gave him the rundown. He requested as I email 
status, to include Dumbacher [Deputy Associate Administrator for Exploration Systems Development], 
Cooke and Ryschevich [NASA Chief Engineer]. 
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5:35am PDT, 8/22/09: MOM called Leonard Hee (LCROSS S&MA [Safety and Mission Assurance]); 
left message on cell phone with status. 


5:45am PDT, 8/22/09: Latest update on fuel consumed is ~ 1 40kg, which leaves us barely enough prop to 
finish mission nominally; Ken Galal [LCROSS Mission Design Lead] and the Mission Design (MD) 
team are in. The team is discussing potential improved responses to this IRU fault since we are about to 
lose contact again at 1pm. 

(Status Email sent) 

6:10am PDT, 8/22/09: Discover VR playback from the more than 7-hours ago was overwritten because 
of the high-level of captured activity/events (thruster firings). 

6:15am PDT, 8/22/09: We are in Dead Band 3 (DB3) positioning accuracy mode right now which is 
looser than our normal DB2 mode. This was done to conserve propellant. 

6:30am PDT, 8/22/09: Thruster 7 thermostat kicked on by itself which shows our thruster thermal control 
system is working fine even under DB3 control - a good thing as it means we can safely keep the 
thrusters warm without requiring unnecessary expenditure of propellant. 

• /*■«»/ 

7:00am PDT, 8/22/09: Duration of geometric outage caused by the normal transition through the plane of 
the ecliptic will begin at 1pm PDT. 

♦ The duration for the outage for commanding is 9h 16mn 

♦ The duration for the outage for telemetry is 8h 2mn 


8:00am PDT, 8/22/09: Estimated propellant remaining is ~60 kg. How much we need to finish the 
mission is a function of what we do with the remaining mission. If we strip most activities out we have 
about a dozen kg margin. 


(Status Email sent) 

8:20am PDT, 8/22/09: Strategizing on the upcoming outage and what mode we should be in through the 
outage. 
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12:00 pm PDT, 8/22/09: Held the command approval meeting and we decided to do the following: 

♦ Do a table upload (update table 95) so that we require a longer persistence of the IRU no-go (should 
it occur - has not since last night), prior to the ACS reacting with fail-over to the Star Tracker rates 
(instead of reacting at 1-sec, we require 5 seconds of persistence prior to switching over). This will 
ignore transient triggers of the IRU. 

♦ Upload a new TMON [Telemetry MONitorJ-RTS 1 [Relative Time Sequence] pair (both verified on 
the S/C simulator) which monitors the fail-over of the IRU to Star Tracker rates, and executes a 
power-cycle of the IRU to attempt to restore IRU functionality, (which is precisely what we did with 
ground commands as we did after this original anomaly occurred early this morning). 

♦ If the IRU recovers we will be back to nominal mission, with typical (small) propellant consumption 

♦ If the IRU does not recover, this recycling of IRU power will occur indefinitely if required. 

♦ While this recycling of power is going on, the spacecraft is balancing between intentionally drifting 
to save propellant, but also using the STA rates to assure we don't tumble, staying power positive 
and avoiding payload instruments look into the sun. 

♦ The key is that in this mode the spacecraft will consume a finite amount of propellant, but not empty 
the tank, which is what we’ve attempted to avoid. 

(Status Email sent) 

2:00 pm PDT, 8/22/09: We have implemented the command approval meeting approach and loaded onto 
the spacecraft. We have just entered our out-of-view period for commanding and will lose telemetry in a 
little while (as expected because of our geometric constraints flying through the plane of the ecliptic). We 
will regain telemetry at ~10:40pm PDT, and regain commanding ability at 1 1 :00 pm. We will meanwhile 
be doing some planning and others will be attempting to get some rest in order to be able to work the 
resumed coverage after the outage. 


(Status Email sent) 

2:35 pm PDT, 8/22/09: Our telemetry has just dropped, as expected, because of the geometric out-of- 
view period for LCROSS (plane of the ecliptic). We have done what we could to minimize the risk to the 
mission while going through this out-of-view period. Now we wait for surfacing on the backside at 10:40 
pm PDT where we regain telemetry and see how we fared through this period. Think good thoughts 
while we continue planning, and some of us get rest. 

(Status Email sent) 
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10:40 pm PDT, 8/22/09: Team is assembled and waiting for AOS 

(Status Email sent) 

10:43 pm PDT, 8/22/09: Have telemetry lock on DSN-45 and DSN-34 

♦ ACS looks good. IRU looks good. There were no IRU events during the outage. Also none of the 
thruster TMONs fired during the outage so temps are good. 

♦ (Dan going offline. John Marmie [Deputy Project Manager] wearing the hat.) 


11:20 pm PDT, 8/22/09: Commanding locked on. This shift will focus on pitch flip to preserve good 
Earth angles on our primary omni antenna. DB3 - no decision yet as to whether we defer to this mode, 
currently in DB2 mode. ACS to look at what it would take to make STA rates flyable (with no IRU). This 
shift will also gather data to build case for root cause. 

11:50 PDT, 8/22/09: Systems estimates 0.07 kg prop usage while out of view prior to pass. Will refine 
prop estimates tomorrow. 

12:00 am PDT, 8/23/09: Updated filter factors for IRU. Dan left for home. 

12:45 am PDT, 8/23/09: Pitch flip successfully perfonned to preserve good Earth angles on our primary 
omni antenna. Comm margin is looking good. 

1:40 am PDT, 8/23/09: Changed downlink data rate to 64 kHz. 

2:30 am PDT, 8/23/09: Begin VR playback. Skipping clock correction. Discussions on criteria for ending 
DSN emergency services. 

(Status Email sent) 

3:30 am PDT, 8/23/09: This morning’s comm pass has cautiously given us renewed hope as all systems 
seem to be performing nominally, barring yesterday’s anomaly resulting in a significant loss of propellant. 
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Commanding locked on to DSN-45, DSN-35 approximately 20 minutes after telemetry lock, as planned. 
We also performed a pitch flip to preserve good Earth angles on our primary omni antenna... our link 
margin is looking good. From telemetry, an estimated 0.07 kg propellant was used during the 9-hour 
comm outage. This pass is intended to last till 3:30 pm PDT...with shift change occurring at 7 am. 

• r««-/ 

4:00 am PDT, 8/23/09: Telecon w/ NG [Northrup-Grumman] to discuss plan forward. NG did brainstorm. 
RTS TMON in place. Next step forward, get ACS to handle the STA rates better. [NG] to send out his 
summarized thoughts. Need to understand how the STA rates are affecting the performance. . .don’t see 
using STA rates as a long term solution. Possibly adjust the rate gains. Also need to keep the thrusters 
wann. [NG] to tweak on the RTS to accomplish more. “We have the patient stabilized.” Let it drift - turn 
ACS on, let it drift - turn ACS off. . .see how it performs. When do we let go of DSN emergency services? 
Jim would want nominal control of S/C if IRU failed. Ideas to focus on short term, mid-term, long term, 
etc. Large group of ACS personnel back. We can change ACS without large SW changes... look at 
changing parameters first. 






5:30am PDT, 8/23/09: Doug Cooke emails wanting to kn ow our intentions. Here’s Marmie’s response: 

“Ooug- 

Currently LCROSS still maintains a positive fuel margin, even though it's been reduced 
considerably. It's our intention to continue on a course that would allow LCROSS to meet full 
mission success. In the meantime, we need to take a dose look at our remaining events and 
refine/prioritize as to what's absolutely necessary in order to conserve fuel. We also still need 
to further investigate as to the root cause. There is much work to do in the short term, but 
for now our intention is to stay the course. 






6:00 am PDT, 8/23/09: Testing out SIMDB-3 mode. Checking to see if thruster temps behave. 






6:30 am PDT, 8/23/09: Dan back on line in the MOCR; John Marmie heading home for sleep. 

(Status Email sent) 






7:15 am PDT, 8/23/09: Shift Change notes: 

♦ No difficulties during the night shift; IRU has had no errors during the night or during the outage 
prior 

♦ Performed omni pitch-flip to improve comm., after the plane of the ecliptic transition (per plan) 
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♦ We remain in our “spacecraft emergency” declaration with DSN and they have been very 
supportive. We are discussing strategies for when we might step away from this declaration, 
understanding it should be as soon as possible. 

♦ Nav is reporting our trajectories look good. The consumption of propellant during the IRU anomaly 
was costly, but the trajectories are holding for the mission. 

♦ Will be holding a discussion with the ACS and technical teams on the topic of the STA rates 
improvement/viability. If there were to be an IRU complete failure, use of the star tracker rates in its 
place would require we improve the quality of the rates by a couple orders of magnitude to get the 
daily prop use back into line with our nominal use. We are evaluating options. 

♦ Prop estimates are being continually re-evaluated 

♦ Next geometric com outage for today is (due to continued transition through the plan of the ecliptic): 

♦ Commanding: 3 pm - 12-midnight PDT 

♦ Telemetry: 3:30 pm - 11:30 p PDT 


(Status Email sent) 


12:31 pm PDT, 8/23/09: spacecraft continues to operate nominally (no IRU anomalous activity). 

♦ We are working a couple different activities along the lines of mid-term and longer-term solutions to 
help preserve propellant. 

♦ I will give an overview of these activities in a future update - right now we are in a command 
approval meeting. 


(Status Email sent) 

r-w/ • r««-/ 


2:45 pm PDT, 8/23/09: We are evaluating a cascading response to addressing a potential for an IRU 
failure to help preserve propellant during a fault. Here’s the logic: 

♦ Step 1 : We monitor for the IRU health for an indication of a problem. If we see a problem we assure 
it is persisting for more than 5 seconds to make sure it is not a transitory issue. If it does not persist, 
there is no reaction taken and the IRU remains the means for rate feedback. If the IRU problem does 
persist, then we power-cycle the IRU as this cleared the problem we experienced on Saturday. This 
process continues indefinitely (with a 12-minute cycle time) if the IRU fault returns after the power- 
cycle (A complete IRU failure handled in Step 2). 

♦ This mode was implemented last night and is flying on the spacecraft right now should an IRU fault 
occur (and no fault has occurred). 
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♦ We are uploading (now) a more robust version which actually uses a PDE-disable [Propulsion and 
Deployment Electronics] for all but 90 seconds of this 12-minute period, greatly reducing the 
exposure to the noisy STA rates which caused the original propellant-consumption problem. 

♦ Step 2: We contend with a completely-failed IRU. We are discussing means of implementation, but 
the idea is to prevent either the IRU (which is presumed faulted in this scenario) or the star tracker 
(which has the noisy rate feedback) from driving thruster firings and expending prop. 

♦ Zero-out the ACS rate loop gain in SIM-DB3 ACS, thereby relying only on position information 
(bang-bang) for spacecraft position control (no rate loop). This could be elegant in that it does not 
require a FSW [flight software] change - just a constants change. 

♦ Note: This does not affect sun point mode (SPM), as that is a different process and would also 
require two failures for this to be a problem: a fault condition which drives us into SPM and a failure 
of the IRU. This mode is in exploration/simulation right now. 

♦ Consider directly controlling when the thrusters are enabled via the PDE. We could judiciously/ 
sparingly employ the ACS on a duty-cycle (similar to Step 1 mitigation), enabling loose control, 
thereby reducing the total propellant consumed (e.g., drift out to 30 deg and then turn PDE control 
back on to get it back into 10 deg). 

♦ What this might do to thruster heating is uncertain; we had been running in SIM-DB3 with 0 deg 
yaw for the past few hours and most of the heaters kicked on at low temps, but some still were 
marginal-low - we have since moved back to -20 deg yaw, per earlier mission thermal validation. 

♦ Step 3: Consider a thruster activity monitor to assure that regardless of causation, excessive thruster 
firings trip a TMON which (could) disable the PDE in an attempt to assure a large unintended 
expenditure of propellant is limited. This will be evaluated after working the earlier steps of triage. 


r-w/ 




10:45 pm, 8/23/09: John M. on shift. 


1 1 :00 pm, 8/23/09: MOM and Flight Team B on shift. Craig Elder (NG Electrical lead) at remote console. 
Marmie in control room; Andrews dialed in. 






11:20 pm, 8/23/09: comm, checks. 


11:30 pm, 8/23/09: beginning of telemetry track on DSS-34 Canberra. No IRU faults. Power nominal. 
Thermal and thruster temps look good. SimDB2 mode, -20 degree yaw. 55 kg-58 kg prop estimated 
remaining. 30 minutes to commanding capability. Less than 0.2 kg/day prop usage estimated during 
outage. Nominal state. Monitoring for IRU fault... allows us to immediately react and work mitigation 
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strategies. Anticipate no maneuvers on this pass. Comm strength looks stable. Some variation was noticed 
on the last pass. 






12: 10 am, 8/24/09: commanding capability lock on DSS-45 Canberra. Switching to 64 Kbps downlink rate. 

(Status Email sent) 


12:30 am, 8/24/09: “LCROSS Update: 

♦ Telemetry/Commanding lock from Canberra was achieved with stable comm strength on schedule 
and as expected. Telemetry indicates no IRU faults occurred during the comm outage. Spacecraft is 
in a nominal state (SIMDB2, -20 deg yaw). Power and thermal temps look good. During the outage 
we estimate that the propellant use rate was less than 0.2kg/day (nominal and consistent with 
expectations). 

♦ During this pass, we will perform a clock calibration, monitor for IRU faults and continue 
development and testing of on-board mitigations. While in view, we will be able to immediately 
react if the anomaly re-occurs. We anticipate no maneuvers on this pass. End of track (pass) will be 
at 4:35 pm PDT. A mission status update 2 will be held at 9:00 am.” 


r-w/ • r««-/ 

2:00 am, 8/24/09: Headed home. All systems still nominal. 

-5:00 am PDT, 8/24/09: Star tracker currents have been moving around, venturing into the yellow zone. 
We pulled up old data from earlier in the mission and are seeing current trending upwards. We also pulled 
up STA rate information from earlier in the mission, and discovered in general the STA rates have been 
getting noisier over the course of the mission. 


7:20 am PDT, 8/24/09: Star tracker currents climbed to near red-limits. We turned off the power on the 
STA because of this; we are flying by IRU. We are evaluating what our drift rate would be while STA is 
off in order to establish a forward plan with minimized use of the STA. NG will be establishing contact 
with the STA vendor to discuss what we are seeing. 

7:30 am PDT, 8/24/09: Canberra is indicating they are having 60 kph gusts and may have to stow all the 
antennae. We are not in view to any other Earth assets at this time. 

(Status Email sent) 
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7:45am PDT, 8/24/09: Galileo Avionica is on two-month summer vacation. We are putting in an 
emergency call with them to get support on what we are seeing. We will be informing LRO of what we are 
seeing [since using the same STA]. 

7:50 am PDT, 8/24/09. Cathy Peddie and Tom Ajluni of LRO are both aware of the STA situation. 

7:55 am PDT, 8/24/09: Winds are dying down at Canberra. 

• /*■«»/ 

8:05 am PDT, 8/24/09: Ops Team B is remaining on shift for a while longer while Team A (who has 
arrived to relieve Team B) is in the MSR [Mission Support Room] working forward scenarios. 

8:07 am PDT, 8/24/09: Canberra reporting winds are 40 kph steady with gusts to 60 kph. 

(Status Email sent) 

8:30 am PDT, 8/24/09: We are in a telecon with Galileo Avionica describing the situation at present. 

11:30 am PDT, 8/24/09: weather at Canberra is sustainable. LCROSS coverage continues. 

We are concurrently working on adjustments to existing TMONs which require STA quaternions so that if 
STA were to drop off while out of view, there would be no problems with the STA being unavailable/failed. 

(Status Email sent) 

12:45 pm PDT, 8/24/09: LCROSS just powered up the star tracker to monitor currents; we have not yet 
initialized it (i.e., we are not yet closing the loop with the STA). 

♦ After speaking with Galileo Avionica (STA vendor), they indicated that we could open our yellow and 
red limits on current before having a concern. They have no problem with us re-powering the STA. 

♦ We have a working theory on what was making the currents change with time and are studying it 
right now. 
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1:15 pm PDT, 8/24/09: Lost telemetry for a period of time due to Canberra stonn - high winds. It was 
restored within a few minutes; no ill effects to LCROSS. 

1:30 pm PDT, 8/24/09: We’ve shared plots of the STA rate noise with the STA vendor and they are not 
seeing anything that they thought was indicative of a failing STA, and have also noted noise of this 
magnitude is normal, and they were not specifically concerned with the data they saw. This infonnation 
coupled with the vendor assessment that we can set our current limits about 100% higher than before 
gives some confidence that we can continue to use the STA. 

(Status Email sent) 

3:00 pm PDT, 8/24/09: Given how recent the STA information is and our limited testing of it, we still 
consider the STA suspect until we can perform more testing when we achieve AOS tomorrow. We have 
tested and are now loading modified TMONs which provide for not having the STA available. The spacecraft 
will drift slightly during this outage, but the drift is expected to be very small over this 9-hour period. 

♦ LCROSS commanding coverage for this pass on 8/24/09 ends at 3:35 pm PDT (right now, due to 
geometric constraints) 

♦ LCROSS telemetry coverage for this pass on 8/24/09 ends at 4: 10 pm PDT (due to geometric 
constraints) 

♦ LCROSS commanding coverage for the next pass on 8/25/09 is 1:10 am -4:10 pm PDT (due to 
geometric constraints) 

♦ LCROSS telemetry coverage for the next pass on 8/25/09 is 12:35 am - 4:45 pm PDT (due to 
geometric constraints) 

(Status Email sent) 

08.24.2009: 1 1 :45pm PDT: Mom and Flight Team B and NG on shift. 

12:35 am PDT, 8/25/09: Lock on Canberra DSS-34 telemetry (on backup station as well). Winds aren’t 
as bad as last night. Valve temps look good, payload temps look good, power looks good. Comm link 
looks good. No maneuvers anticipated. 15 knots, 22 knot gusts. 0.2 kg/day prop usage. 

1:10am PDT, 8/25/09: Lock on Canberra DSS-45 commanding. 

(Status Email sent) 
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1:25 am PDT, 8/25/09: “LCROSS Update: Telemetry/Commanding lock from Canberra was achieved with 
stable comm strength on schedule and as expected. Telemetry indicates no IRU faults occurred during the 
comm outage. Spacecraft is in a nominal state (SIMDB2, -20 deg yaw). Power and thermal temps look 
good. During the outage we estimate that the propellant use rate was less than 0.2 kg/day (nominal and 
consistent with expectations). We are currently flying with the star tracker off due to elevated currents noted 
from telemetry during the last pass. 

♦ During this pass, we will re-power the star tracker and monitor. From the data collected we will 
search for root cause of the elevated current signature. Contingency mitigations will be developed 
and uploaded to offset loss of the STA, in conjunction with IRU mitigations. Optional studies will be 
to re-introduce the 1-Hz clock adjustment to see if it has an effect on currents. While in view, we 
will be able to immediately react if an anomaly re-occurs. We anticipate no maneuvers on this pass. 
End of track (pass) will be at 4:45 pm PDT. 

♦ Current pass planned activities: Pass duration: 12:30 am-4:45 pm PDT. 

♦ Search for root causes of STA (PSE Svc 7) current signature 

♦ Develop contingency mitigations against loss of STA, integrated with IRU mitigations 

♦ Re-power STA for data collection 

♦ Load updated IRU/STA mitigation command products 

♦ Configure for out-of-contact (STA powered on or off) 


• /*■«»/ 

2:15 am PDT, 8/25/09: Canberra predicts high gusts/storm action this morning. Currently have both DSS 
34/45 telemetry. 


2:30 am PDT, 8/25/09: Flight Director Team A walked thru the TMON/RTS interaction on the 
whiteboard before proceeding with STA activation. 

4:15 am PDT, 8/25/09: When going thru procedures to activate star tracker. . .TMON22 failed to load. We 
are attempting to determine why. Think it’s just a “masking” problem. Also debating the risk of losing 
the star tracker if we power it manually, bypassing the TMON. Slight worry that automated threshold 
wouldn’t be activated with a risk of Canberra going down due to high winds. Will most likely wait till 
shift handover to discuss with Flight A. 


(Status Email sent) 
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7:45 am PDT, 8/25/09: We’ve heard that LRO has looked back in their STA current data and are seeing 
some interesting currents as well - addressing them with Galileo as well. We will continue to share notes 
between projects as we have three trackers between us. The LCROSS team is not convinced there is no 
problem with the STA given the relative increases in currents and noise, despite the vendor saying they are 
within limits. We are flying a single STA and we are unclear of its health, so we are moving cautiously. We 
can keep trajectory in pretty good position flying on the IRU alone for a few days without STA correction. 

(Status Email sent) 

10:30 am PDT, 8/25/09: Forward strategy for the day: 

♦ Primary line: Work on a no-propellant-consumed option upon a failure of the IRU 

• The logic is that if we were to fail the IRU, the STA rates, even with our current mitigation in 
place which has the STA rates active for only 90 seconds out of 12 minutes (see previous updates 
on this topic), we would consume more propellant than we want. (It would be nearly an order of 
magnitude better than the original incident, but still too much to be a long-tenn mitigation as we 
could consume all our remaining prop margin during a multi-hour outage.) 

• Our next level of improvement is to update the existing TMON so that with a complete IRU 
failure, we do not go to the STA rates at all and simply go into a propellant-saving drift mode 
until coverage is resumed. This is not a permanent solution but allows us to live to another day 
with no propellant consumed. We believe that our ample power margins will enable this to be 
feasible even through an ~8 hour battery draw, (which is unlikely because we are still power- 
positive while 60 deg off sun). 

• When we AOS, we would then have some decisions to make regarding limited use of the STA 

• This solution would not require a FSW change, which is great. 

♦ Secondary line: Work on understanding the STA condition. 

• We are working with the vendor (Galileo) on this topic and I will capture some of that logic in a 
subsequent note. 

♦ Tertiary line: Work on getting a control law in place which can support making use of the noisy 
STA rates, allowing us to go back to the nominal mission profile. 

• In parallel to the activities above, we have NG working on modifying the control law gains for 
the ACS controller to be able to work with the STA rates, per the baseline design. 

• If we are successful, we could return to the original design basis which is to use the IRU as 
primary rate feedback (as it is right now) and STA as backup. 
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11:37am PDT, 8/25/09: Jay Jenkins [LCROSS program executive] questions about the approach sent in 
the previous email: 

♦ Jay: NG mentioned yesterday that sun point mode relies on the STA for rates. 

♦ Dan: The SPM relies on the IRU for rates, but if the IRU has failed, it hops over to using the STA 
rates. 

♦ Jay: Is this only if the IRU has been flagged as “bad” by the time you are in SPM? 

♦ Dan: Yes, and this is nominal baseline design (i.e., not one of our mitigations). In fact this is what 
led to our problem in the first place: We had a fault on the IRU, which then led the ACS to switch 
over to using the STA rates. With our updating of the required persistence for what turned out to be 
a transitory flag set by the IRU, we already avoid what happened last Saturday. 

♦ Jay: Am assuming you have checked SPM and other autonomy rules for inadvertent reliance upon 
the STA? 

♦ Dan: This is what they are running through right now downstairs in the simulations. As you imply, 
we need to make sure that unintended circular logic doesn't drive us back to where we have a 
problem. 

♦ Jay: If you allow drift, I assume you would be disabling the sun angle rule which would drive you 
into SPM? 

♦ Dan: This is another topic we are discussing. If we enter SPM, we go onto the coarse sun sensors 
(CSS) and the IRU. If the IRU has failed we are back to using the STA rates. From the standpoint of 
minimum success criteria, we could disable the sun angle rule and take our chances in the interest of 
saving propellant. 

♦ Jay: What about the low bus voltage rule which would kick you into SPM? (I recall the low voltage 
sense is pretty conservatively high - so without changing it I don’t know you’d make it 8 hours of 
discharge without tripping into an SPM.) 

♦ Dan: Yet another topic we are also working. There was some debate over whether or not we wanted to 
leave this in place (after assuring the voltage levels settings are appropriately set) since if we were truly 
at a very low voltage, we’d be needing to get on the sun anyhow. The rub is how you do it. Using the 
STA rates (as is) will be a costly way to bias back on the sun in this condition. ... In the end, we believe 
that for a multi-hour outage, this is academic as we have arguably 7-9 hours on the battery, and this 
duration presumes the moment we have LOS, the IRU faults, and we quickly get off sun by more than 
60 deg (power-negative state). We actually would likely be power-positive for some time, buying us 
many more hours. If we have some sort of spin, we may average power-positive anyhow... . The end 
game is to get our control law tuning where it can work with the noisy STA rates, enabling our mission 
to move forward as designed. Covering these multi-hour outages (not multi-day outages), we should 
be fine until the modified controller is in place. . . . If we cannot get a control law in place that can work 
with the STA rates, then we’ll need a more brute-force solution. 
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♦ Jay: Clarification: Am assuming this plan assumes continued emergency status for limiting your 
black out period (basis of your power assessment)? 

♦ Dan: It does for now. We have been entertaining some interesting options about having some low- 
level monitoring for our transmitter signal on a continuous basis (could be other than DSN assets) 
and have the team ready on-call to support as needed. This would be a longer-view strategy once we 
get past this near-term triage. 

(Status Email sent) 

1:00 pm PDT, 8/25/09: We are powering up the star tracker to perfonn some monitoring of the currents. 
We will do this while the STA is disabled (not in the control loop), but it lets us confirm some theories 
about the currents we are seeing. This work is being done in parallel with the activities in the MSR 
working towards the drift protection mode upon an IRU failure. 

(Status Email sent) 

r-w/ • r««-/ 

2:15 pm PDT, 8/25/09: We have successfully powered up the STA and have enabled it back in its 
nominal mode (closing the loop on the STA). A current-monitor TMON is in place, per recommendation 
from Galileo. IRU is still the source of rate data. IRU continues to be faultless since the Saturday start of 
the anomaly. The persistence protections for the IRU fault flag remains in place. Meanwhile, we just held 
a command approval meeting (CAM) for our drift mode contingency TMON & RTS. This is the scenario 
in which we monitor the IRU fault condition and if unable to recover a faulted IRU, ACS is turned off 
and the spacecraft drifts instead of expending propellant based on STA rates. We decided we are close 
but need to have more time to have eyes on it to assure there are no unintended interactions, given the 
time left on this pass. 


3:30pm PDT, 8/25/09: Star tracker assessment 

♦ Galileo says the star tracker is fine 

♦ No evidence of an anomaly according to them - the currents are not an indicator of bad behavior; all 
housekeeping data is fine as well. 

♦ Our estimate of what is happening: Our STA current levels changes (which we never saw before) 
could be attributed to a clock time correction factor we added into the system to assure ground 
station clocks and spacecraft clocks are in sync. This correction factor was applied shortly before we 
started to see the STA current response. 

♦ Recall that the LCROSS STA clock is updated via a SW sync pulse over the 1553 bus while the STA 
current is measured at a time relative to the hardware 1PPS signal from HKIO [housekeeping input/ 
output board]. So, when a clock correction factor is applied to the SW clock, the sampled current 
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from the STA is moving in phase: i.e., it is not measuring the current from the same place in the STA 
operational cycle, and the current levels change based on what the STA is doing functionally. 

♦ So if our premise is true, we should see currents changing with time along prescribed levels 
matching the STA EIDP [end-item data package] oscilloscope snapshot... 

♦ To prove this premise, we enabled the STA, left the time correction factor in play, and recorded the 
currents over time. Sure enough, the currents started changing and after we waited enough time, and 
traced like an o-scope, the plots right out of the STA EIDP. 

♦ This is difficult to describe without seeing the plots, so the plots can be made available; however, it 
is the smoking gun for the strange currents and appears to confirm the STA is behaving as it should. 

♦ We still need to address the STA rate noise as a backup for a failed IRU (per the baseline design). As 
noted in a previous email, we are working this as well by attempting to create a controller design 
which can work with the noisier signal. 


• /*■«»/ 

4:00pm PDT, 8/25/09: We are entering LOS with the star tracker powered and enabled, per nominal plan, 
given the discussion below. We have installed an additional current monitor on the STA, per Galileo’s 
suggestion to further protect the unit. We’re starting to get a handle on our tigers. 

08.26.2009: 1 :30am PDT: MOM and Flight Team B and NG on console, John Marmie dialed in. 

1:40am PDT, 8/26/09: Lock on Canberra DSS-34 telemetry (on backup station as well). Star tracker 
looks good. All looks nominal. Commanding in a half hour. 

2:10am PDT, 8/26/09: Lock on Canberra DSS-45 commanding. 

(Status Email sent) 


2:00 am PDT, 8/26/09: LCROSS Update: 

♦ Telemetry/Commanding lock from Canberra was achieved with stable comm strength on schedule and 
as expected. Telemetry indicates no star tracker or IRU faults occurred during the comm outage. 
Spacecraft is in a nominal state (SIMDB2, -20 deg yaw). Power and thermal temps look good. During 
the outage we estimate that the propellant use rate was less than 0.2 kg/day (nominal and consistent 
with expectations). We are currently flying with the star tracker on, fully integrated with ACS. 

♦ During this pass, we will closely monitor the star tracker and IRU and conduct further thermal 
behavior studies in SIMDB3 (Solar Inertial Mode Dead-Band mode 3) at -20 deg yaw. Recall that 
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SIMDB3 has “looser” dead-hand control, thereby conserving more propellant. Data from the ACS 
monitoring will allow us to derive a new controller for star tracker rate-based control. We anticipate 
no maneuvers on this pass. End of track (pass) will be at 4:50 pm PDT. 

♦ Current pass planned activities: Pass duration: 1 :40 am - 4:50 pm PDT. 

♦ Dwell and observe thermal response of thrusters, taking appropriate actions on TMONs to ensure 
heaters trip on their own. 

♦ Monitor progress of ACS to derive a new controller for STA rate based control. 

♦ Shift handover starting at 7:00 AM Pacific, 14:00 UTC. 


• /*■«»/ 

2:45 am PDT, 8/26/09: There is a concern that the study requested in DB3 (provided in the plan from 
shift A to B) will cost a lot of prop if we return to DB2. The transition from DB3 to DB2 costs about Vi 
kg, (or 3 days of prop, in current DB2 - Per PSE). SE [systems engineering] is objecting. We may not do 
the DB3 study till we believe the cost/benefit was fully considered. 

1:00 pm PDT, 8/26/09: The spacecraft continues to perfonn nominally. The STA, which was enabled last 
night before we went out of view, is behaving well and is in full operation. As previously noted, we have 
implemented a current monitor on the STA, per the manufacturer recommendation. 

The IRU continues to operate without any trips of the flag that started on Friday (and we have the 
persistence check in place to prevent Saturday’s events from occurring again). 

We have an updated prop chart which shows that we have 50 kg of prop remaining, with an uncertainty 
of +/-12 kg. If we streamline the remainder of the mission, we estimate we will need 26 kg of propellant 
left to finish a lean mission (but still meet full mission success). 

(Status Email sent) 

2:10 pm PDT, 8/26/09: The ACS team at NG is conducting a very detailed review of a new controller for 
handling the noisy LCROSS STA rates. The estimates for prop use are coming in around the 1-2 kg per 
day range which is considerably better than the 150 kg/day rate during last Saturday’s anomaly. This 
would be a very good interim solution in advance of an even more refined solution. Recall this only 
comes into play when there is a permanent failure of the IRU. 

5:10 pm PDT, 8/26/09: We have gone back out of view. Spacecraft continues to operate uneventfully/ 
normally. 
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♦ STA rates retuning: We have received the updated tuning gain constants from NG following an 
exhaustive 3+hour review. Preceding this were many hours of hi-fi simulations on a dynamic 
simulator, which was discussed in the review. We have now begun doing simulations on the NASA- 
ARC spacecraft simulator, as well as checking the existing TMONS to assure everything is set up 
right. This verification process may take two days since we have to also monitor the spacecraft. 

♦ Estimated prop usage in the event of an IRU failure: 

• This retune effort: 3-4 kg/day 

• Currently-flying mitigation: -24 kg/day 

• Original (anomaly) prop usage: -150 kg/day 

• We anticipate a further improvement with another iteration, but will evaluate priority of seeking 
additional savings versus other risk mitigations. 

♦ DSN coverage: In order to relieve some of the stress of the LCROSS DSN Declaration of Emergency, 
tomorrow (Thu) will be the last day we request both primary and redundant DSN assets. Starting 
Friday we will drop to only primary assets, which halves the number of assets tied up with LCROSS. 

(Status Email sent) 


2:00 am PDT, 8/27/09: LCROSS Update: 

♦ Telemetry/Commanding lock from Goldstone/Canberra was achieved with stable comm strength on 
schedule and as expected. Telemetry indicates no star tracker or IRU faults occurred during the 
comm outage. Spacecraft is in a nominal state (SIMDB2, -20 deg yaw). Power and thermal temps 
look good. During the outage we estimate that the propellant use rate was less than 0.2 kg/day 
(nominal and consistent with expectations). We are currently flying with the star tracker on fully 
integrated with ACS. 

♦ It is anticipated to be relatively quiet on night shift as we will just be monitoring spacecraft systems. 
We anticipate no maneuvers on this pass. End of track (pass) will be at 5:25 pm PDT. 

♦ Current pass planned activities: Pass duration: 1:20 am - 5:25 pm PDT. 

♦ Spacecraft was configured for out-of-view. 






2:30 am PDT, 8/27/09: John M. sign off. All systems still nominal. Just performing VR playback. 

(Status Email sent) 
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11:00 am PDT, 8/27/09: All LCROSS systems remain nominal 

♦ We are working on two prop-saving contingency threads right now to enable moving away from the 
Emergency Declaration with DSN. 

♦ ARC Center management and ITA (independent technical authority) are strongly in support of an 
approach which halts prop consumption (free drift), upon excessive thruster firings or other faults, 
while out of view of the spacecraft. 

♦ We are working on two variations of free-drift responses upon a fault, with the detailing being what 
we trigger on, and how we enter the free-drift conditions. 

♦ The bulk of the team is focused on this activity now with a small crew upstairs monitoring the health 
of the spacecraft. 

♦ Our secondary activity is to complete the simulation on the new controller tuning gains for making 
use of the STA noisy rates. We have this running on the spacecraft simulator in the background, 
while we work the free-drift options above. 

(Status Email sent) 

8:32 pm PDT. 8/27/09: As noted in previous email status on LCROSS, we are working on prop-consumption 
detector as an outer protection means for future prop-hungry failures/anomalies. We spent all day walking 
through several different options and working the details, including initial sims. 

♦ Exec Summary: We use our existing safe mode, augmented with a thruster firing monitor to see if 
we are expending a lot of prop, and then if so, invoke our familiar spin mode from our existing 
Centaur leak mitigation sims, and then power off the ACS to stop all prop consumption. 

♦ The Details: (as best the PM can illustrate): 

• We create a Thruster Firing Monitor (TFM). (We believe one exists via virtual recorder 3.) 

• We create a TMON which compares the TFM count to an identified threshold level. If the 
threshold is breached then we have excessive thruster activity and we invoke prop saver mode 
(PSM). The setting of this threshold value needs to low enough to limit the amount of prop used, 
but high enough that we are not prone to false triggers. 

• We utilize the existing, proven sun point mode (SPM) as is. This includes the existing feature 
that when this mode is invoked, LCROSS turns on the transponder to “phone home.” 

• We add two additional entry points into SPM: 

• IRU failure. This skips the step where it tries to bring up the STA rates. It will first go to SPM 
and then if the IRU is truly failed, will be protected from excess prop use by the TFM below. 

• TFM TMON. If the number of thruster firings allowed within a given period of time is exceeded, 
we are expending too much prop and we go into SPM. 
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• Once in sun point mode (SPM): 

- Something had to have faulted to be in this mode. This mode and the entries into this mode all 
exist and are previously tested plus the IRU and TFM fault conditions described above. 

- If the IRU is fine (i.e., we entered this mode for reasons other than an IRU fault), there is no 
need to do anything else; we use the pre-existing, standard SPM activity to save fuel. 

- If the IRU has faulted (and that is why we are in SPM in the first place), the control system 
will attempt to use the STA rates. Until we get the tuning updated on this mode (and NG has 
put together a solution) this mode will lead to excessive thruster firings; however, our TFM 
will kick us into PSM. 

- If TFM limits are exceeded, even while in SPM, we will be kicked into PSM. 

• Prop saver mode (PSM): When triggered, we borrow what was learned during the Centaur leak 
handling studies and invoke a low-prop-cost spin on the spacecraft to stabilize it prior to turning 
off the LCROSS Attitude Control System (ACS) and go into a free coast until ground command 
intervenes. Once the ACS is turned off the spacecraft uses zero prop . Our goal is to define a spin 
rate that maintains a stable attitude for 9+ hours in free coast. We still go into a prop-free mode, 
but this would keep us power positive and the payloads safe until ground intervention. 

r-w/ • r««-/ 

1:00 am PDT: Lock on Goldstone DSS-24 telemetry. All looks nominal. Prop still 58. Temps look good, 
battery, STA, IRU, no ACS faults, CSS looks good, 23.5 deg off sun., payload heaters working, no alarms. 

1:20 am PDT: Lock on commanding DSS-34 Goldstone commanding. 

(Status Email sent) 


1:40 am PDT, 8/28/09: LCROSS Update: 

♦ Telemetry/Commanding lock from Goldstone was achieved with stable comm strength on schedule 
and as expected. Telemetry indicates no star tracker or IRU faults occurred during the comm outage. 
Spacecraft is in a nominal state (SIMDB2, —20 deg yaw). Power and thermal temps look good. During 
the outage we estimate that the propellant use rate was less than 0.2 kg/day (nominal and consistent 
with expectations). We are currently flying with the star tracker on fully integrated with ACS. 

♦ It is anticipated to be relatively quiet on night shift as we will just be monitoring spacecraft systems. 
We anticipate no maneuvers on this pass. End of track (pass) will be at 5: 15 pm PDT. 

♦ Current pass planned activities: Pass duration: 1 :00 am -5:15 pm PDT. 

(Status Email sent) 
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11:45 am PDT, 8/28/09: The spacecraft continues to perform nominally. LCROSS IRU and STA remain 
in the nominal configuration/usage on the spacecraft. 

♦ We are currently pouring through test plans to verify the prop saver mode mitigation. Our focus is to 
try to make sure we are doing no harm, but accomplishing the goal of saving prop. 

♦ We are also studying options of using NASA Ground Network (GN) to enable LCROSS to reduce 
the DSN coverage requirements; the thinking is that this team cannot support extended 24/7 
coverage without experiencing crew fatigue, thereby growing a different risk. Once the PSM is in 
place, we could potentially use GN assets to simply listen for LCROSS “phone home” signals and 
then notify the LCROSS team to mobilize. This will allow the team to safely get rest during future 
unattended operations. 

(Status Email sent) 


4:20 pm PDT, 8/28/09: spacecraft ops team continues to monitor the spacecraft today. The spacecraft is 
performing flawlessly in normal control cruise mode. The team has been working two tasks today: 

♦ DSN sep and braking burn rehearsal (Scheduled long ago and important to keep in place). 

♦ Finished the test plan for the verification of the prop saver mode (PSM) midday. We have since been 
conducting that test plan in the MSR. 

♦ There is a chance we’ll be able to hold a command approval meeting (CAM) tomorrow (Saturday) 
prior to an upload the same day. We’ll see how the verification goes... 

♦ DSN coverage is now 24/7 as we are no longer geometrically constrained by being in the Southern 
Hemisphere. We have to replan staffing of this small team, accordingly. 

♦ Tonight we will be having a monitoring mode with minimal staff while the bulk of the team gets 
needed rest. There is no scheduled commanding of the spacecraft. 

♦ We will then start up tomorrow with nearly the whole team on day or swing shift supporting what 
could be an upload and monitor of the PSM. 

(Status Email sent) 


8:48 am PDT, 8/29/09: Susan Kurtik from DSN @ JPL has stated the following: 

♦ The fire chief reported that the fire [in the Pasadena hills near JPL] continues to burn, but at a 
slow rate with current winds at a low level. The lab continues to be closed to all but essential 
personnel. 

♦ Normal DSN tracking operations were conducted overnight. An ops chief and data control engineer 
supported operations in parallel from the DSN Remote Operations Center (ROC) in Monrovia to 
ensure preparedness in the event that the essential personnel must leave the lab. 
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♦ We followed Chandrayaan-1 late yesterday, but there was no communication from the 
spacecraft. We are awaiting word on the mission status from the ISRO [Indian Space 
Research Organisation] agency. At this time, they have not requested additional DSN support. 
The emergency status for the LCROSS spacecraft is expected to continue through at least 
DOY [day of the year] 243. 


(Status Email sent) 


12:02 pm PDT, 8/29/09: The separation and braking burn DSN rehearsal/ERT [engineering readiness test] 
was conducted successfully yesterday. This was principally a DSN activity with the LCROSS FD [flight 
director] and FC [flight controller], who ran the rehearsal from the LCROSS SOC [science operations 
center], in order to enable the MOCR [mission operations control room] to continue to monitor the actual 
spacecraft telemetry. 

♦ Just completed the command approval meeting (CAM) for the PSM. It was approved. We are 
targeting an afternoon upload to the spacecraft. 


8.29.2009: 12:00 pm PDT: MOM and Flight Team A on console on shift. Andrews in control room. 

1:50 pm PDT, 8/29/09: Beginning of upload of command products and new RTSs for monitoring. Go for 
start of RAM load command plan. 


(Status Email sent) 


4:45 pm PDT, 8/29/09: “The new RTS and TMONS have been loaded into RAM and currently disabled. 
We are testing the functionality on the simulator before activation. We are also scanning through the 
thruster firing packets to correlate analytical/empirical data. After simulations are complete and analytical/ 
empirical data is correlated, we will load the RTS and TMONS in EEPROM then activate. The activities 
are taking longer than expected. . .but we hope to have completed tonight sometime. 

♦ We will monitor the rest of the evening and early morning. Tomorrow we will monitor the S/C and 
simulate the ACS new gain algorithms. 

♦ Fires are within a mile of DSN. They are still staffed, maintaining control and if need be can 
transfer control to the Monrovia facility. 






277 


Appendix D — Timeline Notes from Propellant Anomaly 


6:50 pm PDT, 8/29/09: Simulations continue, but some problems they’re working out. Analytical/ 
empirical data correlated. 


(Status Email sent) 

1 1 :00 pm PDT, 8/29/09: We have uploaded and activated the modified and new TMONs and RTSs which 
implement the new prop saver mode (PSM). We took an incremental approach to loading and activating 
them watching each step of the way. We are now in watching mode, and will be throughout the night. We 
will have minimal coverage during the early morning hours as that proved to be very effective for a few 
to get some good rest. We will do this again in the early hours tomorrow. 

♦ Our steps for tomorrow (Sun) are to complete verification/simulations of the updated tuning 
parameters for the STA rates mode accommodation, while the crew continues to monitor the 
spacecraft with the new parameters in place. If we complete this task we will have the command 
approval meeting on Monday morning for potential uploading on the spacecraft. 

♦ This has been another good day - methodical progress towards a more robust remainder of the 
mission. 


(Status Email sent) 


9:10 pm PDT, 8/30/09: spacecraft continues to perfonn nominally with the new PSM in place. We have 
been in monitoring mode all through the night. 

♦ JPL/DSN continues to be a worry as the fire has grown considerably; however the latest 
reports are that JPL’s threat is lowering a bit. They have fully prepared with their back-up 
facilities in Monrovia and have stated they do not expect a lapse in coverage. 

♦ As noted in previous status update, today we are moving into verifying the control system re -tuned 
gains to be able to accommodate the STA rates as-is. We are optimistic about this solution greatly 
reducing the prop consumed, as we have verified it on a hi-fi simulator at NG; however, we need to 
do the modal checks up here at NASA-ARC on the spacecraft simulator. The command approval 
meeting is tentatively scheduled for Monday morning if all goes well. 


12:45 pm PDT: ACS simulations are now showing the spacecraft maintaining attitude. Cut and paste 
error in one of the tables was found. 


3:00 pm PDT: Testing is going well. Debate is to have an ASR (Activity Selection Review), decision 
point for all these products, then a CAM saying they’re done. . .then execution. CAM may not be done by 
morning. If we don’t do CAM in the morning we don’t take advantage of full day. One option is to do 
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CAM on Monday. . .exec on Tuesday. Decisions on how to package/implement for the first time. Team is 
determining the best schedule for assembling the command plan and holding the CAM. 

(Status Email sent) 


3:45 pm LCROSS Status: 

♦ All is fine this afternoon. Prop, thennal, power, ACS, IRU, star tracker, DSN are all in a nominal state. 

♦ ACS testing of new gain settings is going well in simulations, but takes time. Team is determining 
the best schedule for assembling the command plan and holding the CAM (command approval 
meeting). . .which possibly will not happen until tomorrow. 

♦ As of this morning, 7:20 am PDT, the wildfires have consumed -35,200 acres and are -5% 
contained. DSN personnel do say the danger to JPL is reduced. 

♦ A few fortunate team members and their families did get to enjoy a few hours last night at the 
Foothill College observatory where they viewed the Lunar South Pole, Jupiter and its moons, and 
view an Iridium flare. It was good positive energy tha nk s to Brian Day, the LCROSS E&PO 
[education and public outreach] lead. . .and a reminder just how exciting this mission is. 


4:50 pm PDT: Full Shift A + Planning Team be in at 0800 PDT Monday for the CAM for Mod-SIM DB2 
rates. 


5:00 pm PDT: Would like to continue to run simdb2 mod overnight. Sims show that it uses no more fuel 
than the quiescent state. Have an 8 am ASR to discuss ramifications of new mode. Draft of command 
plan in place. . .then hold a CAM. 

6:15 pm PDT: Shift handover. Last 24 hours monitoring only. Yesterday evening uploaded prop saver 
products. S/C configured for out of view. New products are enabled. Handover to station DSS65 complete, 
we have them until 1 : 15 in the morning. NG will monitor tonight till midnight allowing NASA-ARC team 
to rest. Jim Strong will come in at midnight with second person arriving [for buddy system], SIMDB-2 - 
20yaw. Prop steady at 0.2 kg/day. Temperatures nominal. Near tenn plan: modified simdb2 simulations 
running overnight. 

(Status Email sent) 

6:40 pm LCROSS Status: All systems remain nominal. The last 24 hours have been in monitoring mode 
only. Handover to Madrid DSS65 has now been completed. In order to allow the NASA-ARC team to have 
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a short break, Northrop-Grumman will begin monitoring until midnight, with the NASA-ARC ops team on 
call. NASA-ARC ops will then resume at midnight and monitor thru the night. ACS simulation of the 
modified SIMDB-2 mode will continue to run overnight... initial results are looking promising. Working 
with DSN on getting single (no backup) continuous coverage through DOY244 (today is DOY242). 


6:40 pm: NG on station until midnight. 

8:10 pm PDT, 8/30/09: All systems are nominal; spacecraft is in normal mode with IRU and STA active. 
There have been no additional IRU faults since the original anomaly. 

Our estimated daily prop use under the new tuning is estimated to be an amazing 0.06 kg/day when in 
IRU rate mode - higher when using STA rates usage. We are still verifying functionality on the NASA- 
ARC spacecraft simulator. 

(Status Email sent) 

9:40 am PDT, 8/31/09: All systems are nominal. We are continuing verification on the spacecraft simulator 
of the re-tuned control system gains and developing command products to enable their uploading. This 
could happen as early as tomorrow following a command approval meeting as early as Tue morning. The 
team is strategizing with Center Management and Center ITA when we will have the products in place to 
support release of Spacecraft Emergency. 

3:10 pm PDT, 8/31/09: DSN reports, “The JPL laboratory will re-open on Tuesday, 1 September. NISN 
[NASA Integrated Services Network] reports that no AT&T assets used for DSN network communications 
are threatened by the wild fire.” 

(Status Email sent) 

4:35 pm PDT, 8/31/09: Completed the review of the products for the control system tuning gain updates 
addressing the noisy STA rates. The prop-performance of the new rates is impressive: 

♦ While on IRU rates, prop use drops from ~0.2 kg/day to -0.06 kg/day (yes, two zeros). This is 
notable because this means our normal/non-fault prop usage will drop, enabling us to bank 
additional propellant as the mission goes forward. While on STA rates, prop use drops from -150 
kg/day, to -1.6 kg/day. This is the improved usage from the anomaly scenario. It is still too high to 
sustain through the end of the mission (should the IRU fault), but with the outer-loop prop saver 
mode monitor, we can limit how long we stay in this condition (if we ever enter it). 

(Status Email sent) 
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8:15 am PDT, 9/01/09: Today we will have our command approval meeting (CAM) for the new tuning 
constants, presuming the uploading products were completed last night (the guys aren’t in yet). If this goes 
per plan, we could have the new tuning parameters on the spacecraft by the early afternoon. We are 
discussing validation plans, but clearly we want to do this while we have the core team in play during day 
shift and swing; we then go into pure monitoring mode during late-swing and graveyard shift, where we’ll 
just watch and monitor until the morning. We continue to walk a balance between continuing to move the 
mitigations forward and assuring crew fatigue does not become a major factor. 

(Status Email sent) 

• /*■«»/ 


12:30 pm PDT, 9/01/09: We completed our CAM on the updated tuning gains for LCROSS control, 
which included reviewing all the dynamic responses of the system through any of the mode changes. The 
plots look really good. We are now breaking for lunch and then will begin the process for uploading the 
parameters onto the spacecraft and conducting some initial testing later. 

(Status Email sent) 


4:50 pm PDT, 9/01/09: We have just concluded our upload of the new tuning constants on the spacecraft 
and have tested the system performance in both IRU-rate mode and in STA-rate mode. We have 
confirmed stability in that the controller properly bounces off the expanded deadbands in both modes 
(i.e., it is controlling). We will be remaining in IRU-rates mode, with the new tuning constants, all 
through the night. This will be monitored continuously, and of course recorded to give us a large data set 
from which we can look at prop usage estimates. Tomorrow we will likely spend more time in the STA 
rates mode to better confirm prop usage, but this will be evaluated tomorrow. . . . Another good day. 

(Status Email sent) 

• /*■«»/ 


7:30 am PDT, 9/02/09: LCROSS behaved nicely last night while staying in IRU-rates mode with the new 
tuning parameters. Initial estimates are that the prop rate of consumption (measured over 6 hours) was 
0.05 kg/day. The simulations showed 0.06 kg/day so this is good correlation, and it is notably better than 
with the previous tuning parameters which consumed 0.18 kg/day. It is important to remember that 0.18 
kg/day is completely sustainable through the end of the mission, so this savings which comes from the 
new tuning parameters allows us to bank additional propellant reserves over time (if all were to continue 
nominally). Recall that the reason we changed the tuning parameters, however, was not to improve the 
IRU-rates prop consumption, but that of the STA-rates prop consumption. Last night we merely validated 
on the spacecraft that the STA-rates mode is stable, but we have not been in this mode long enough to 


281 


Appendix D — Timeline Notes from Propellant Anomaly 


effectively assess the prop use and compare to simulations. We will be talking today about how and for 
how long we do this. 


(Status Email sent) 


2:15 pm PDT, 9/02/09: We started our STA-rates prop testing 1.5 hours ago. We were only going to test 
for an hour, but are finding the prop use is coming in better (lower) than estimates. The axes plots are also 
showing nice behavior, so we extended the testing a while further to better characterize its settling 
position. The current prop cost in STA-rates mode is 0.4 kg/day. 

♦ LCROSS will be cancelling this Saturday’s planned Earth look. The originally scheduled activity 
was expected to cost ~4 kg of propellant as designed. We are discussing a greatly simplified “stare” 
at the Earth that would be similar to a star look (i.e., no slewing back and forth across the moon, 
which is what drives up the prop numbers). Such a simplification would cost < 0.5 kg prop. 
LCROSS Science believes conditions are ok for performing this over the next couple of weeks. 

♦ This is an example of future activities that we will be evaluating as a function of our available prop. 

(Status Email sent) 


3:50 pm PDT, 9/02/09: DSN has informed LCROSS that the Odyssey mission will go into Spacecraft 
Emergency if they do not keep their pass between 1:50 pm - 2:20 pm PDT on Thursday. This means 
Odyssey requires an asset that LCROSS was going to use under its Emergency Status for this 30-min 
period. LCROSS has agreed to give up this time for Odyssey and will assure the LCROSS spacecraft is 
accordingly prepared. 


(Status Email sent) 

4:20 pm PDT, 9/02/09: We have just completed our extended testing of the STA-rates mode in order to 
assess propellant usage while in this mode. The daily estimated usage of the STA-rate mode yielded 
between 0.7 — 1.3 kg/day, which is in line with the simulated estimates. We have had our command 
approval meeting for the next test which will be conducted later today. We will be testing on the 
spacecraft the telemetry monitor (TMON) which monitors thruster firing and compares it against a preset 
limit value. This is the heart of the prop saver mode (PSM). We will have disabled the RTSs which would 
actually take the spacecraft into sun point mode; we are just confirming (as we have already on the 
simulator) that the spacecraft logic performs as intended. 

(Status Email sent) 
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7:20 am PDT, 9/03/09: Last night the TMON which performs the prop saver mode monitoring/tripping 
was successfully validated on the spacecraft. The monitor was able to watch thruster firings and compare 
those to a threshold value. When the thruster count exceeded a pre-defined value, the TMON tripped and 
attempted to activate sun point mode and then free-drift, but we blocked that activation as SPM was 
previously verified and free drift is something to avoid if we can. We are presently scheduled to exit 
LCROSS spacecraft Emergency Declaration with DSN today at 5:00 pm PDT. I met with ACD [NASA- 
ARC Center Director] and ITA this morning to walk through our verification and validation status. We 
have a couple questions left to address, but it is our expectation this Emergency Declaration will expire 
per plan. We have worked with DSN to assure that they can address our forward ConOps plan which is to 
not have out-of-view lapses for more than 9 hours at a time. DSN has been able to secure this schedule 
through Sunday and is currently working plans beyond that. 

(Status Email sent) 

4:10 pm PDT, 9/03/09: We are starting our third and final Earth “lap” today. LCROSS remains in good 
shape. We are currently flying on the IRU with the enhanced tuning parameters, per the plan. We 
executed an omni pitch-flip, per the plan, required every time we pass through the plane of the ecliptic in 
order to get our omnis properly positioned with good link margins. This required maneuver costs 0.5 kg 
in total to execute, as we have allocated in our budgeting (and it is a required move). We are configured 
for our Com outages which begin tonight as LCROSS enters into out-of view periods not to exceed 9 
hours at a time, per the modified OpsCon plan. 

(Status Email sent) 

5:00 pm PDT, 9/03/09: LCROSS officially drops “Spacecraft Emergency” declaration. 

LCROSS received concurrence from NASA-ARC ITA that the controls/mitigations in place had been 
sufficiently verified and validated to support our modified ConOps plan which provides for no more than 
9 hours out of view without checking in with the spacecraft. 

We are discussing what additional testing and simulations may be required to support going longer periods 
of time while out of view, but there are no immediate plans in place to extend the out-of- view period. DSN 
has successfully provided LCROSS with a < 9-hour out-of-view period through Sunday and is working on 
the weeks that follow right now. DSN understands our ConOps constraints. LCROSS has been evaluating 
other potential monitoring means such as NASA GN, and to that end has been holding telecons on this topic 
with the appropriate parties. The degree to which GN assets will be needed is of course a function of how 
successful DSN is in securing assets (and so far has been very successful), and the effort required to verify 
and validate each station for LCROSS use. There is a fair amount of set-up required to enable these other 
means of monitoring. Tonight will see an 1:15 hr and a 9 hr outage before morning. We are configured in 
nonnal mode using IRU rates and burning nominally 0.05 kg/day propellant. 
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(Status Email sent) 

5:25 am PDT, 9/04/09: LCROSS achieves AOS following a 9-hour out-of-view period. Consumption of 
prop typical at -0.04 kg/day. We’re getting back on track! 

(Status Email sent) 

12:30 pm PDT, 9/04/09: Given that LCROSS has made it through this current storm and has rescinded its 
declaration of spacecraft emergency with DSN, I will be ceasing these semi-hourly updates on LCROSS. 
Our work is of course not done as we continue to navigate our risk for the remainder of the mission, but 
if LCROSS experiences another major anomaly as we move forward in our last 30 days of this mission, 
I’ll resume updates like these so the LCROSS stakeholder community has timely information of our 
activities to address a major problem. 

As a final thought, I wish to express how impressed I am by this LCROSS team. It’s said that the real 
salt of a team shows when they are thrown into an adversarial situation - one where they are 
struggling to make it, and it is there where you see what they are made of. Well, with what I saw from 
this team over the past two weeks, I’m a pretty fortunate project manager. Thanks to the stakeholders 
for your support and patience See you at the moon. 


End Notes 

1. RTS = Relative Time Sequence. A RTS is invoked by ground command or trigger event. ATS = Absolute Time Sequence. 
An ATS begins when the spacecraft clock reaches the specific ATS start time. 

2. Mission Status Updates were held by the MOM with PM in attendance. Local NASA-ARC stakeholders would attend in 
person and other stakeholders would dial-in for a video-telecon. MOM would status activities initially every morning and 
then less frequently later. 
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